evolve-distill

Files (1)

• SKILL.md

SKILL.md (first lines)

# evolve-distill — a proven workflow becomes a skill, safely

This is evolve's answer to autoskill-style skill creation
(`docs/evolve-plan.md`): distill a *successful multi-step workflow*
into a reusable skill — but behind every evolve gate. The difference that
matters: autoskill runs an LLM subprocess and **auto-writes** the skill to
disk. Here **you are the extractor** (the agent-is-deriver pattern), and the
output is a *proposal*, not a skill — it lands only after an eval and a human
say so. That prevents the sprawl and poisoning that ungated auto-creation
invites.

Paths: `EVOLVE=${CLAUDE_PLUGIN_ROOT}/scripts/evolve`; resolve state with
`node $EVOLVE/honcho_client.ts status` (honors `XDG_DATA_HOME`).

## When NOT to distill

Most sessions produce nothing distillable, and that is the correct outcome.
A workflow earns a skill only when **all** hold — otherwise stop:

- **Reusable class, not a one-off.** A procedure for a *class* of tasks
  ("set up a new FastAPI service"), never a single task ("fix today's bug").
- **≥2 witnessing sessions** (the anti-overfit bar — autoskill's headline
  failure is single-trajectory skills that bake in run-specific paths). The
  exception is an explicit user ask ("make a skill from this"), which is its
  own evidence.
- **Non-obvious.** It saved real work and isn't common knowledge or a
  single command.
- **Not already covered** by a hub skill or an existing overlay (check
  `overlay.ts report`; near-duplicate → propose a *patch* via evolve-review,
  not a new skill).

## Procedure

**1. Find candidates.** Recurring signal points at what to read:

```bash
node $EVOLVE/overlay.ts distill-candidates
```

Task classes seen in ≥2 distinct sessions, with their session-ids. Thin by
design — it says *where to look*, not *what to create*. No candidates? Say
"nothing to distill" and stop (run `digest.ts --backfill` first if the
journal is empty).

**2. Read the evidence — you are the extractor.** For a candidate, open the
named sessions' transcripts (under the projects dir from `status`) and confirm
the *same* multi-step workflow actually recurred and succeeded. Two sessions
that merely touched the same skill are not a workflow — read them and judge.
Verify success from artifacts (git log, test output) where the transcript is
ambiguous; never trust a self-report of "it worked."

**3. Distill class-level, strip the session.** Draft the SKILL.md body as
*When to use / Core principles / Workflow* with concrete tool commands.
Ruthlessly remove anything specific to one run — fixed paths, values, repo
names, calibrations. Whatever only one trajectory needed goes to a
`references/` note, not the body. A session-specific name is an automatic
reject; name for the class (`setup-fastapi-service-local`, never
`fix-x-today-local`).

**4. Write the eval — no eval, no registration.** A skill with no way to
prove it works never enters the catalog (the catalog is trusted recall). Draft
a bench scenario that asserts on **artifacts**: `{"id": ..., "prompt": ...,
"assert": "<shell that exits 0 iff the skill worked>", "judge": "<optional
rubric>"}`. This is the gate — `overlay.ts propose` refuses an
`overlay-create` without it.

**5. Stage the proposal (never write the skill).** Pipe the proposal to
`propose` — it validates the eval and the ≥2-session bar, then stages to
`pending/`:

```bash
echo '{"kind":"overlay-create","name":"<class>-local",
       "description":"Use when …","body":"## When to use…\n## Workflow\n1. …",
       "summary":"distilled <class> workflow","signal":"recurred in N sessions",
       "sessions":["<id1>","<id2>"],
       "eval":{"id":"smoke","prompt":"/<class>-local …","assert":"…"}}' \
  | node $EVOLVE/overlay.ts propose
```

**6. Record the mining (symmetric mining, Law 9).** Bank what *worked*, not
only failures: `honcho_client.ts observe --type technique --target agent
--content "[technique] project=… — <the reusable method>"` so the next
distill pass sees the reinforcing signal.

**7. Hand off.** The proposal sits in `pending/`; `/evolve-review` presents it
for approval, and approval replays it via `overlay.ts apply-pending` →
**GR4 skill-write scan** → overlay scaffolded at confidence **0.0** with its
`bench.json` bundled. It earns trust only through `overlay.ts record` outcomes;
never present a fresh distilled skill as proven.

## Hard rules

- **The house standard applies to generated skills too.** Read
  `skills/writing-skills/references/skill-authoring.md` before proposing one.
  A pipeline whose output is exempt from the standard reintroduces every
  problem the standard exists to prevent, only faster. In particular: the
  description carries trigger conditions and never a workflow summary (D2), and
  the proposal is not complete without a negative-activation scenario proving
  the skill stays silent on its nearest neighbour (E2).
  **Known gap — the tooling cannot carry that scenario yet.** `overlay.ts
  propose` requires every scenario to bring an `eval.assert`, and an
  `expect_no_activation` scenario has none by design: it is decided by a trigger
  probe, not an assert. An accepted proposal also writes a single eval object
  where `scripts/skill-bench.ts` expects a spec with `skill` and `scenarios`.
  Until both schemas change together — its own change, not a patch here — author
  the negative scenario into `tests/bench/<name>.json` by hand, and never read
  "propose accepted it" as evidence that E2 was met.
- **De-identification reuse test (R1)**: after stripping case-specific
  entities, ask whether this user would need the same policy again — if what
  remains is generic advice, distill nothing. Empty is the default success.
- **Never invent steps.** Only workflow steps the sessions actually
  demonstrate; single-trajectory details go to `references/`, not the body.

- **Propose, never write.** This skill stages proposals only. Applying is
  `/evolve-review`'s human-gated step; scaffolding runs the poisoning scan.
- **≥2 sessions or an explicit ask.** One impressive session is the overfit
  trap, not evidence.
- **No eval, no registration.** Enforced mechanically at `propose`.
- **Class-level names; hub skills are read-only.** A distilled skill is always

From uhstray-io/huhhb, MIT.

More from uhstray-io/huhhb