learn

Files (1)

• SKILL.md

SKILL.md (first lines)

# Learn: harness retrospective + wiki knowledge harvest

The close-out retrospective, in two phases:
- **Phase 0 (process):** diff how the engagement was EXECUTED against the discipline the
  skills/hooks prescribe, find where the operator (you) or the harness drifted, and fix the
  harness so the next box does not repeat it. This is the "improve the harness on it" half.
- **Phases 1-7 (knowledge):** read the WHOLE finished engagement, diff it against the existing
  wiki, and promote only the DELTA as durable, generic knowledge. The safety net that catches
  everything not captured live during the box.

The client-data boundary is enforced by code, not by this prose: engagement-derived
content reaches `wiki/` ONLY through `wiki-stage.py` -> `wiki-promote.py`, which runs
`check-leaks.sh` and fails closed. This skill READS the engagement and WRITES only
generic knowledge; it never edits the engagement's own files.

## When it fires
- At close-out: run `Skill(learn)` once the engagement is marked
  `## STATUS: SOLVED` (OWNED/ROOTED/COMPLETE) AND its `walkthrough.md` is assembled
  (the CLAUDE.md execution-loop close-out step).
  It self-clears the moment this skill writes `<eng>/.learn-done`.
- On demand: "extract learnings", "what did we learn", "harvest lessons", "distill",
  "post-mortem into the wiki" - the same steps apply.

## Phase 0: Process retrospective (harness self-improvement) - DO FIRST

Before harvesting knowledge, ask: **did we execute the way the skills/hooks told us to, and
where we did not, whose fault is it - mine, or the harness's for not catching me?** The point is
not self-flagellation; it is to turn each drift into a concrete harness change so the next
engagement cannot repeat it.

### 0a. Diff execution against discipline
Re-read the skills that governed this engagement (`ctf-box`, the `hunt-*` used, the CLAUDE.md
execution loop) and check each mandated step against what actually happened on disk:

| Discipline it prescribes | How to check it was (not) done | Common drift |
|---|---|---|
| Recon tooling complete (nmap AND ffuf AND nuclei-read for web) | `ls targets/$ENG/recon/*.png`, `.recon-tools` marker, tmux window names, `log.md` | ffuf never run; nuclei launched but output never read |
| Screenshot EVERY finding as it lands | count deliberate `poc/*.png` vs findings in `Killchain.md`; `.screenshot-nudged` | shots only at the very end; transient states lost |
| Wiki-first before exploiting each fingerprinted service | wiki queries in transcript / `log.md` | jumped to exploitation from memory |
| A hook nudge fired -> was it acted on? | grep the transcript for a nudge (e.g. "switch to ffuf") whose action never followed | nudge ignored under momentum |
| State-first / capture-as-you-go | `state.md`/`loot.md` updated mid-box vs all-at-end | prose-in-chat lost |

**If the engagement ran under the campaign driver** (`bb/pt/ctf-workflow`), pull its own numbers -
they say precisely where the driver fought the operator, no guessing:

```
python3 scripts/campaign.py --eng <ENG> ledger --json
```

`drift` = network calls the driver never emitted (improvisation off the board); `dry_rounds` /
`reframe_lenses_used` = how hard the campaign had to reframe to find anything; `paused_hosts` = hosts
that banned us; `board.dead` vs `board.closed` = the exhaustion-to-finding ratio. A high drift or a
long lens list is the retro's starting point.

For each drift, name the **root cause** honestly: a *skill* that under-specified, a *hook* that
should have fired a reflex but did not (or fired and was ignorable), or a *me* failure the harness
had no mechanism to catch.

### 0b. Turn each drift into a harness change
Route each finding to its fix TARGET and make the change (small+clear -> apply now with the
normal edit+test loop; larger -> record as a proposed item for operator approval):
- **Reflex gap** (a capture/coverage signal the harness could have caught) -> add/extend a hook
  in `skills/hooks/` (stay within "reflexes = capture/route", NOT methodology), with a test in
  `tests/`. Run `python3 -m pytest tests/ -q` green before moving on.
- **Discipline under-specified** -> tighten the governing skill (`ctf-box`/`hunt-*`): make the
  step an explicit ordered checklist item, not buried prose. Skill bodies carry lesson PROSE +
  `[[wikilink]]` ONLY - never inline runnable exploit/rootkit/PoC code (magic-signal dumps,
  syscall PoCs, CVE exploits, reversing walks). That code is the KNOWLEDGE half: promote it to a
  `wiki/` page through the Phase 5-6 gate and point the skill at it with a one-line "try X (see
  [[page]])". A skill body loads wholesale into model/API context on EVERY invocation, so raw
  offensive code there trips the Claude API safety classifier (real: ctf-box's LKM-rootkit +
  CVE-PoC lessons got flagged on load, forcing a manual purge) and bloats context. Prose that
  says WHAT to try in WHAT order belongs in the skill; the code that does it belongs in the wiki.
- **Script/analyzer gap** -> extend `scripts/` (coverage, next_move, find-lint) to surface it.

### 0c. Log the retrospective (generic, tracked)
Append a dated section to `docs/superpowers/harness-retro.md` (create if missing): the drifts
found, their root cause, and the harness change made or proposed. This doc is about the HARNESS,
so it is generic and tracked - describe failures generically ("on a web box ffuf was skipped"),
never with client host/IP/cred.

**Naming the engagement in this tracked file:** use its dir name ONLY when that name is a neutral
codename (`<platform>_<boxname>`). Many engagement dirs are named after the target itself (a host, a
product, a client short-name) - writing one of those into `harness-retro.md` publishes a client
marker, and `check-leaks.sh` derives its marker list from `targets/` dir names, so it WILL fail the
gate. When the dir name is target-derived, describe the engagement generically instead ("a Symfony
API target", "a single-host web bug-bounty campaign") and name nothing. Run `bash
scripts/check-leaks.sh` after writing the retro; a FAIL here almost always means the dir name leaked.

### 0d. Fill the engagement's agent eval (per-engagement, gitignored)
`targets/$ENG/eval.md` has two halves - auto (real data) and judgement (you). Scaffolded upfront
from `_eval.md` for pentest/bugbounty; for ctf it self-creates the moment the next step
(`eval_metrics.py --write`, already run once by close-out.py at SOLVED) writes to it, so it may not
exist yet if you are running this by hand before close-out fired:

**Auto first - REAL numbers, no estimation:**
```bash
python3 scripts/eval_metrics.py $ENG --write
```
This injects a `## Metrics (auto)` block from the hook telemetry (`.events.jsonl`) + the session
transcript recorded in `.metrics.json`: exact skill / hook / tool call counts, auto-detected drift
signals (every scope-guard block, state-discipline nudge, leak-warn), the start->finish time delta +
idle-filtered active time, and the box's token usage (windowed to its active period). NEVER hand-type
these - the transcript is ground truth (a real box's true output was ~12x an earlier hand-estimate).

**Then the judgement half - only you can write it:** in the same file, fill the Drift moments (what
the auto drift-count can't say: WHY each drift happened + the one-line fix), What went right, and the
Scores. The auto block counts drift; you narrate its cause. Dead-ends come from `Deadends.md`.
Unlike the retro (generic, tracked), eval.md is per-engagement and may name specifics (gitignored `targets/`).

Only after Phase 0 is done, proceed to the knowledge harvest below.

## Steps (Phases 1-7: wiki knowledge harvest)

### 1. Resolve the engagement
```bash
ENG=$(cat targets/active.md)
TYPE=$(grep -m1 engagement_type targets/$ENG/state.md | cut -d: -f2 | tr -d ' ')
```

From Encod3d-Sec/TORCH, MIT.

More from Encod3d-Sec/TORCH