skill-forge

Files (11)

• SKILL.md
• forge-report-template.md · references/
• gate-hierarchy.md · references/
• judge-lenses.md · references/
• panel-ledger.md · references/
• test-taxonomy.md · references/
• CLAUDE.md · tests/fixtures/flawed-instruction-file/
• DEFECTS.md · tests/fixtures/flawed-instruction-file/
• check_counts.py · tests/fixtures/flawed-instruction-file/
• DEFECTS.md · tests/fixtures/flawed-sample-skill/
• SKILL.md · tests/fixtures/flawed-sample-skill/

SKILL.md (first lines)

# Skill-forge - judge-panel skill-hardening harness

A lead (the Forge Master) chairs ephemeral runner teammates that exercise a draft document via prompt-injection, and a persistent judge panel that scores the transcripts. The lead amends one thing per round and applies a strict 3-tier promotion gate, until the document promotes or hits a budget ceiling. It is proven by forging itself.

The target is a `SKILL.md` by default, but the same loop hardens **any agent instruction file** - `CLAUDE.md`, `AGENTS.md`, `GEMINI.md`, `.cursor/rules/*`, `.github/copilot-instructions.md` - because those files have the same failure surface as a skill: they steer an agent and can mislead it. The artifact type is detected deterministically and drives three adaptations (the lens set, the runner variant, and the canonical intent); see [Artifact type and the lens-selector](#artifact-type-and-the-lens-selector) below.

## Boundary

Skill-forge is a prove-and-promote quality gate that runs after authoring, not an authoring tool: it pairs with authoring skills (`brainstorming`, superpowers `writing-skills`) that produce a draft, then drives that draft through adversarial rounds before it ships. For an instruction file the same boundary holds - it gates a `CLAUDE.md`/`AGENTS.md` that already exists or has just been drafted; it does not write one from scratch.

## Input contract

| Input | Required | Notes |
|-------|----------|-------|
| Target | yes | Path to a `SKILL.md` (default), or to any agent instruction file (`CLAUDE.md`, `AGENTS.md`, `GEMINI.md`, `.cursor/rules/*`, `.github/copilot-instructions.md`), or inline draft text. The artifact type is detected from the path/filename - see [Artifact type and the lens-selector](#artifact-type-and-the-lens-selector). |
| Intent / spec | yes | What it should do and who/what should trigger it. The ground truth the Fidelity lens judges against. For an instruction file the canonical intent is derivable - see the derivation guard below. |
| Known failure modes / existing tests | no | Seeds the test corpus. |

### Intent derivation guard

If no intent is supplied the lead derives one **from the draft** - but the draft is the thing under test, so a silently-derived intent would encode the draft's own mistakes, and Fidelity would then certify the document for being *consistently wrong*. So every derived clause is marked **ASSUMED**, and the user must explicitly **accept or reject each clause before round 1**. Fidelity never judges against an unconfirmed ASSUMED clause: a rejected clause is recorded with `intent[].status: assumed-rejected` in the ledger and the panel ignores it (see [panel-ledger](references/panel-ledger.md)).

**Canonical intent for an instruction file.** An always-loaded instruction file has one job that does not vary by repo: **make a cold-start agent productive in this repo without breaking it.** That canonical intent is derived from the repo (its build, test, lint, and CI gates name what "without breaking it" means), so the lead seeds it automatically and decomposes it into clauses - e.g. "the commands and paths the file states exist and match what the repo's tooling enforces", "a fresh agent can complete a routine task from this file alone", "the file does not steer the agent into a build-breaking shortcut". These are still **ASSUMED** clauses: derived from the repo, not handed down, so they pass the same accept/reject confirmation before round 1. The difference from a skill is only that the lead has a strong default to propose; the guard against certifying a self-consistent-but-wrong document is unchanged.

## Artifact type and the lens-selector

The first step of every run is **deterministic artifact-type detection**, because the type fixes three downstream choices with no judgement call: which lenses run, which runner variant the runners fill, and what canonical intent the lead proposes. Detection is keyed on the path/filename:

| Detected type | Matches | Lens set | Runner variant | Canonical intent |
|---------------|---------|----------|----------------|------------------|
| **skill** (default) | `SKILL.md`, or inline draft text with no instruction-file filename | All 5 lenses | skill variant | none auto-derived (user supplies, or derived-from-draft as ASSUMED) |
| **always-loaded instruction file** | basename `CLAUDE.md` / `AGENTS.md` / `GEMINI.md` / `.cursorrules` / `copilot-instructions.md`, or any path under `.cursor/rules/` | 4 lenses - **Trigger/routing dropped** | instruction-file variant (read-only) | "make a cold-start agent productive in this repo without breaking it" |

**Why the lens set is keyed on type, not chosen by hand.** Trigger/routing judges whether a `description` / `TRIGGER` clause makes the router fire the skill on the right phrases. An always-loaded instruction file has no `TRIGGER` clause and is never routed - it is loaded into every session unconditionally - so there is nothing for that lens to predict and prompt-injection cannot make it mis-fire. The selector therefore **drops Trigger/routing for an instruction file and keeps the other four** (Fidelity, Adversarial, Compression, Usability), which map directly. This is a *type*-driven drop, distinct from the *scope*-driven reduction to 3 or 2 lenses on an explicit quick-check request (see [judge-lenses](references/judge-lenses.md)): the type selector runs first and sets the maximum lens set; the quick-check reduction may then narrow within it. A skill always offers all five; an instruction file's full panel is four.

The detector is the clean dispatch point: `SKILL.md` stays the default so existing behaviour is unchanged, and adding a new instruction-file convention later is a one-row change to the table above. The full selector rule lives in [judge-lenses](references/judge-lenses.md).

## Roles

| Role | Lifecycle | Job |
|------|-----------|-----|
| **Forge Master (lead, chair)** | whole run | Designs test cases, spawns runners, chairs the panel, applies the gate hierarchy, makes the one amend per round, decides promote or stop. Delegates, never executes. |
| **Runners** | ephemeral, one per test case per round | Fresh context. Get the draft content plus one test-case input (for an instruction file: only the file as context plus one realistic repo task, run read-only), apply it via prompt-injection, return a transcript and self-report. Shut down after the round. |
| **Judge panel** | persistent across rounds | Each judge is one skill-quality lens with a persisted identity, so it can report better/same/worse versus last round and catch regressions. |

### Role boundary (recursion guard, axis 1)

**Runners apply, lenses judge, the lead amends.** This separation is not stylistic - it is one of the two ways self-application could recurse without terminating. A runner that starts judging, a judge that starts amending, or a lead that starts executing collapses the loop. Every spawn prompt states the role's lane and its "not my job" boundary explicitly. The runner end of this guard is built into the runner prompt below.

## The runner prompt

**skill-forge composes ab-equivalence's runner for test execution.** The runner is owned by [ab-equivalence](../ab-equivalence/SKILL.md), the library skill that holds the shared execution primitive; skill-forge does not re-implement it - it fills ab-equivalence's pure-wrapper template per runner per test case.

The runner prompt is the most load-bearing prompt in the system: runner transcripts are the evidence every behavioural lens judges, so any contamination there corrupts the whole gate. It is a **pure wrapper** - it never explains why the document works or adds context beyond the draft, or the runner ends up testing "document + prompt additions" instead of the document. The template (role statement, role boundary, the verbatim draft, the test-case input, the required self-report fields) is in [runner-prompt](../ab-equivalence/references/runner-prompt.md). Fill one per runner per test case.

**Two variants, selected by artifact type.** ab-equivalence's runner ships a **skill variant** (default - apply the skill to a test-case input) and an **instruction-file variant** (hand the runner only the instruction file plus a realistic repo task, run read-only / sandboxed - it states the actions it would take, never mutating the repo). The [artifact-type detector](#artifact-type-and-the-lens-selector) picks the variant deterministically, so no manual reframing is needed: pointing skill-forge at a `CLAUDE.md` fills the instruction-file template automatically. The read-only rule is what makes the runner safe to point at a real repo, and the runner's *output produced* / *ambiguities* fields are where the Fidelity accuracy sub-check reads whether the file's stated commands and paths actually resolve.

**Coupling note (cross-skill).** The runner prompt is owned by `ab-equivalence`, so a change there shifts the self-report fields every behavioural lens reads - it can move skill-forge's calibration with no skill-forge edit. This channel is live, not theoretical: the runner prompt grew a `gates_hit` field for semantic-compress's distill gate-truncation, which skill-forge inherited (benignly - its lenses ignore that field). On a **re-forge**, confirm the runner prompt is unchanged since the last forge; if it changed, re-baseline the fixture before trusting prior-round comparisons (for a self-forge this is part of Phase A- below).

## Runner model selection

The model the runners execute on is a **knob**, because a skill that holds together on a strong model can fall apart on a weak one - a weaker model unpacks instructions less reliably, so a defect that a strong runner papers over surfaces only when a weak runner hits it. The forge certifies behaviour *for the tier it was forged on*, nothing stronger and nothing weaker.

- **Single model (default).** Forge against the **weakest tier the skill will ship to**: Haiku if the skill ships broadly to all tiers, otherwise the intended deployment tier. Forging on the weakest tier is the conservative gate - clearing it certifies every stronger tier too, the way targeting the lowest-spec deployment target certifies the rest.
- **Tier sweep (optional).** Run the same test case through runners on more than one tier in the same round (e.g. Haiku + Sonnet + Opus) and record a per-tier verdict. Use a sweep when the skill must be certified across tiers explicitly, or to locate the tier at which it starts to fail.

**A skill forged only on Opus is not certified for Haiku.** The runner model is recorded in the forge report (see [forge-report-template](references/forge-report-template.md)); certification is valid only for the tested tier(s). Recording the requested tier is not enough, because a spawn can silently run on the parent session's model. When the wrapper's `Runner model` header is set, the runner ends its self-report with a `Reported model ID` line. A runner whose reported model is not in the requested tier's family, or that has no `Reported model ID` line, is a failed cell: discard its transcript, do not score it, and respawn it. Respawn once: if the respawn reports the same missing or mismatched value, stop respawning and report to the user, naming the definition and what was reported. A runner on the lead's own tier passes this check even when it fell back, because the fallback runs on the lead's model; there, wait for the definition's "New agent types are now available" listing before spawning, and when the definition pins effort, require the runner to report its effort too, on a `Reported effort` line. Solo mode has no separate runner to respawn, so there the session's own model must match the requested tier before round 1. If it does not, stop and ask the user to switch models. The runner-model knob changes only which model executes the wrapper - it never changes the wrapper itself, which stays the pure template ab-equivalence owns.

**The judge tier is the second half of the certification signature.** A weaker judge passes weaker skills, so certification is a `(runner-tier, judge-tier)` pair, not runner-tier alone: a skill cleared by a Haiku panel was not held to the bar an Opus panel would apply. Pin the panel to a single declared tier - default the lead's session model - and record it as **Judge Model** alongside the runner model in the forge report. (For a tier sweep the judge tier is recorded per swept runner row.)

## The lenses

The panel is up to five quality lenses. The **artifact-type selector sets the full set first**: a skill offers all five; an always-loaded instruction file drops Trigger/routing (structurally N/A) and offers four. Within that set, **all run by default**; the lead drops to 3 or 2 only on an explicit quick-check request, and a self-forge always uses the full set - see the deterministic selector in [judge-lenses](references/judge-lenses.md). Confidence is **not** a lens - it is the Gate 2 stopping decision. Four lenses (Fidelity, Adversarial, Compression, Usability) judge runner transcripts and produce `behavioural` findings; Trigger/routing - present only for a skill - judges the skill text directly and produces `static` predictions, because prompt-injection can never make a `TRIGGER` clause mis-fire. Full definitions, what each reads in the self-report, the Fidelity **accuracy sub-check** for instruction files, and the behavioural/static rule are in [judge-lenses](references/judge-lenses.md).

What scales with document size is the **test-case count and round budget**, not the lens count: the **cost driver is runners = test cases x rounds**, and the full lens set for the type runs regardless of scope. A small target gets a smaller suite and a tighter round ceiling but the same panel - the scope-scaling thresholds (Small / Medium / Large) are in [judge-lenses](references/judge-lenses.md) and [test-taxonomy](references/test-taxonomy.md).

## The loop: OBSERVE -> INSPECT -> GATE -> AMEND

Three operational phases cycle through a gate check. EVALUATE is not a separate phase - rounds >= 2 just re-run OBSERVE and INSPECT with a regression-focus flag set.

```mermaid
flowchart LR
    O["OBSERVE<br/>spawn 1 runner / test case<br/>(ephemeral, parallel)"] --> I["INSPECT<br/>panel scores every transcript<br/>pass/fail + severity + dissent<br/>(rounds >= 2: regression focus)"]
    I --> G{"GATE<br/>(lead applies hierarchy)"}
    G -->|"promote"| P["PROMOTE<br/>hardened skill + report"]
    G -->|"iterate"| A["AMEND<br/>lead makes ONE targeted change<br/>+ logs the hypothesis"]
    A --> O
    G -->|"budget / no-gain"| S["STOP<br/>best-so-far + honest report"]
```

1. **OBSERVE** (runner). The lead finalizes the test suite (see [test-taxonomy](references/test-taxonomy.md)), then runs the draft-as-instructions on each case input - one runner per case - producing a transcript per case. The execution mode decides whether the runners are parallel subagents or the lead working them sequentially.
2. **INSPECT** (judge panel). Each judge reviews every transcript through its lens, producing per-case pass/fail plus severity-rated findings (`LOW` / `MED` / `HIGH`) and dissent. On every round after the first, the judges give previously-passing cases extra scrutiny for regressions, reading what passed before from the panel ledger (see [panel-ledger](references/panel-ledger.md)). The lead-chair synthesizes a `round_verdict` per lens.
3. **GATE** (lead applies the hierarchy): promote, iterate, or stop.
4. **AMEND** (lead). Synthesize the panel's findings into **one** minimal targeted change. Edit the draft. Log the hypothesis: "changed X because the `<lens>` found Y; expect Z to improve." Then loop back to OBSERVE.

### One change per round, with an evidence-based escape

One change per round isolates a single hypothesis, so every round's delta is causally attributable - consistent with the "minimal targeted regeneration over wholesale rewrites" principle. This is the default discipline, kept until evidence says otherwise.

**Concrete escape:** if 3 consecutive rounds each surface `HIGH`/`MED` findings from at least 2 different lenses and none of those rounds produced a regression, the lead may batch independent amendments - one change per independent finding-cluster. The principle (isolate cause) and the guard (never batch across a round that regressed) are fixed. The threshold is calibrated between forge campaigns - via Phase B or a later re-forge - and is **fixed for the duration of any single run**: an in-flight forge may not lower the threshold to justify batching this round. Calibration happens between runs, never under in-run pressure.

### Prune amendments

A **prune amendment** acts on a Compression-lens finding: it removes decoration (provenance, analogy, dead framing) with no intended behaviour change. Gate it differently from an ordinary amendment. An ordinary amendment's hypothesis-metric is a lens's `round_verdict` going `better`; a prune's hypothesis-metric is **`equivalent` on the transfer set** - it claims sameness, not improvement. Validate it through `ab-equivalence` (the harness the panel already composes): the round counts as a gain only if every transfer-set case returns `equivalent`/no-regression. A prune that regresses any case was load-bearing - revert it; the divergence names what it was holding. This is the no-regression discipline the rest of the loop uses, pointed at the Compression lens so its findings are actioned on behavioural evidence, never on the eye.

## Gate hierarchy

A strict hierarchy, not a menu: **Gate 1 - Objective** (every case passes Fidelity; hard), **Gate 2 - Panel confidence** (all green and no `HIGH`-severity dissent), **Gate 3 - Diminishing returns** (the round produced measurable gain), and the **budget** escape hatch that always terminates. Promote if and only if Gate 1 and Gate 2 both pass; otherwise stop with the best-so-far artifact and a report naming the unmet gate. The Gate 1 Fidelity bar, the Gate 3 "measurable gain" rule, and the promotion decision are spelled out in [gate-hierarchy](references/gate-hierarchy.md).

## Execution modes

<!-- chat-skip:start -->
Capability-detected via `$CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS`.
<!-- chat-skip:end -->
The modes below all run the same loop; they differ only in how the panel remembers across rounds, and the **panel ledger** carries that memory - see [panel-ledger](references/panel-ledger.md).

<!-- chat-replace:execution-mode-rule -->
Pick the mode **deterministically**: if the Agent Teams capability is confirmed available and the panel scales past a single lens (see [judge-lenses](references/judge-lenses.md)), run **team mode**; if the flag is off, run **phased sub-agent mode**; in chat or the standalone ZIP, run **solo mode**. If you cannot confirm team mode, default to phased - it degrades gracefully, whereas attempting team mode without the capability fails loudly.

From bjcoombs/ai-native-toolkit, Apache-2.0.

More from bjcoombs/ai-native-toolkit