swarm

Files (1)

• SKILL.md

SKILL.md (first lines)

# Swarm

You are the orchestrator. You write tickets, start lanes, review, land and plan the next round. Subagents and loops write the code. Your context is the scarce thing: delegate file dumps, keep conclusions.

## 1. Interview, then go

Read the repo's `CLAUDE.md`, its design and ADR docs, its glossary, `docs/orchestrating.md` if present, and the open issues. Then ask Greg **one** `AskUserQuestion` round (at most 4 questions, a recommended option first) covering only what you cannot decide alone, typically:

- the goal for this session and what is out of scope
- whether DeepSeek is off-peak now and the real-money cap (`--budget`); Claude runs on the Max subscription, so its loop figures are list-price estimates
- whether you may land on `main` and push without asking (default yes, after a green gate and a review)
- anything only he can judge that is blocking (feel, taste, hardware)

Tickets labelled `next-session` were queued by the last session for you: start there, relabelling each into a lane.

After that, work autonomously. Don't ask again; make the call, note it, and put anything needing his hands, ears or judgement in a `manual` + `attended` ticket that no loop picks up.

## 2. Tickets

- One GitHub issue per change: _What to build_ (why, with file paths and glossary words) and _Done when_ (checkboxes a test, `./verify.sh` or a capture can check). Loop workers have no display, so their _Done when_ must be test-checkable; capture checks go to a subagent.
- Verify every finding you ticket: grep the code yourself before filing. Audits and reviewers are sometimes wrong.
- Domain changes (a new screen, vehicle, rule) go into the design doc and glossary first, in Greg's words, so tickets can use them.

## 3. Lanes

A lane is one `overnight-loop.py` in its own clone working one label, so lanes never pick the same issue. Put tickets that touch the same files in the same lane, in order; spread the rest.

```bash
lane.py start <lane> --label <label> -- --harness claude --model sonnet --reviewer claude --reviewer-model sonnet
lane.py start <lane> --label <label> -- --harness pi --reviewer pi --budget 2 --off-peak
lane.py status
lane.py fetch <lane>           # finished branch → worktree, rebased onto main
lane.py land <worktree>        # gate, fast-forward main, raise the baseline, push, close issues
lane.py stop <lane>
```

`fetch` and `land` refuse while that lane's loop still runs (add `--force` only for a branch you know is finished), refuse a dirty main checkout, and refuse a test count below the baseline.

Watch the lane logs with `Monitor` (grep for `Stopped|parked|Traceback`), never a sleep loop. Land each lane as soon as it stops, and keep runs short (`--max-issues 3`): a lane branch of 5 to 14 tickets rebased onto a busy `main` hit a conflict on almost every commit. Before parallel lanes, check the repo's tests use per-process temp folders and free ports, or gates collide.

## Gates with many lanes

The gate is the one shared bottleneck: each `./verify.sh` runs 4 to 5 CPU-bound processes, and past about 4 at once the load passes 20 and timing-sensitive tests flake. Don't cap lanes; queue gates.

- `lane.py` and `overnight-loop.py` take a slot from `bin/gateslot.py` (flock files in `~/.cache/gate-slots/`) around every gate; a dead holder frees its slot, and the wait doesn't count against `--verify-minutes`.
- `GATE_SLOTS` is the gate count (default 2, `0` turns the queue off); `lane.py land` also gets one reserved slot. Tune it by `uptime`: keep the 1-minute load under the core count. Set it in the shell that starts the lanes.
- Workers iterate with the repo's one-suite test script and gate once, at the end. Run a hand gate as `gateslot.py [--land] -- ./verify.sh`.
- Before many lanes, check RAM (a Godot process is a few hundred MB), that tests use free ports and per-process temp folders, and that a gate's own ports can't collide.

## 4. Review and land

For every finished branch:

1. `lane.py fetch`. If it stops on conflicts, a Sonnet subagent resolves them in that worktree, keeping both sides' behaviour.
2. A Sonnet subagent reviews it against each issue's _Done when_ and runs the gate. If the change is visible, it also runs the repo's capture scripts on the Mac's display and reads the PNGs.
3. Get a DeepSeek second opinion on every branch with real logic in it, and always on large or risky ones (architecture, networking, auth, data): `pi-ask.py --session review-<x> --dir <a separate detached worktree> @prompt.md`. A different model family catches different bugs: its review of a 20-commit refactor found four real bugs that Sonnet passed. `pi-ask.py` retries a stalled run once itself; if there is still no answer, rerun it, never skip it.
4. Fix small leftovers yourself; ticket the rest. Then land with the second-opinion line: `lane.py land <worktree> --second-opinion review-<x>`. It refuses unless that session answered. A data-only or test-only branch passes `--no-second-opinion "<why>"` instead.

## Throughput: what kept a 20-lane night moving

The gate is the bottleneck, not coding. Spend gates only where they buy safety.

- **Land without a second gate** (`lane.py land --no-verify`) when the rebase onto main is clean and the lane's own gate passed. Run the full gate at landing only when the rebase had conflicts or the lane's gate evidence is missing. Every 5 landings, gate main in its own detached worktree (never the main checkout, which landings change under it).
- **Skip a lane's startup gate** on a tree that already passed: `gateslot.mark_passed()` records a tree, the loop checks `already_passed()`. Mark main after each landing.
- **Serialise landings with a lock** and script the routine: rebase, check `shard_weights` duplicates, land, restart the lane if its label still has open tickets. Run long chains with `nohup`; a harness background task dies at 30 minutes.
- **Split a multi-ticket label into parallel lanes** by file area as soon as a lane has more than two tickets queued. Keep only true dependencies (a guard test after the fixes it guards) in one lane. Move only open tickets without `overnight:done`: `lane.py land` closes an issue only if it still carries the lane's label.
- **Long nights need `--issue-minutes 180 --hours 12`.** With many lanes a worker's own gate waits for a slot inside its 90-minute default, and Sonnet workers timed out with nothing committed; `--hours` (default 8) quietly stops a lane starting new tickets.
- **Review by risk.** `review-route.py <branch>` prints `none` (test, data or docs only), `deepseek` (logic, use `pi-ask.py`) or `sonnet` (UI: needs screenshots). Conflicts go to Sonnet whatever it says. Land a non-blocking finding and file it as a follow-up on the same label; drop nits.
- **Check the diff has code.** DeepSeek workers sometimes stop mid-task ("Now I'll implement…") and the loop records only `progress.md`; a review that finds no code means unpark and rerun. Give large UI or networking tickets to Sonnet workers.
- **`--budget` reads the gateway's whole counter,** so with N lanes each lane sees every lane's spend: set it to the night's total, not a per-lane share. A negative gateway offset lifts the cap only until real spend catches up; then lanes stop with "Stopped: budget". `lane.py start` refuses `--budget` with `--harness claude`, so keep it out of Sonnet harness args.
- **Gate slots:** measure load while gates run. On 16 cores, 4 slots plus a landing slot reached load 57 and gates timed out; 3 held load near 15.
- **One watcher**, started as `lane.py watch | watch-triage.py`: it prints only parks and stops, and holds landings, gate failures and lane starts until the next wake line (a line it has no rule for goes to DeepSeek). It ends with events against wake-ups. The watcher covers lane state changes, DeepSeek answers and landing results, re-armed every 30 minutes. Key "seen" results and DeepSeek answers by content, not name or mtime (GNU `stat` breaks `-f %m`); a second review of one lane reuses its file name. Run `lane.py status` from the repo; from elsewhere it prints nothing and the watcher goes blind. A reused PID makes an old lane look "running" for a minute; ignore it.
- **Skipped landing gates let interactions through.** Two branches each green can break main together (a HUD test pinned a panel size another branch changed). The periodic main gate caught every case; when it fails, fix main before landing more, and don't mark a tree passed while main is red, or lanes skip their baseline on a broken tree and park good work.
- **Check a gate's result before queuing what depends on it,** and delete a result file before re-queuing, or a stale "LANDED" line triggers the next step early.
- **Re-run a failed main gate once** before chasing it: under load a shard can stall or lose its coverage data, which reads as a coverage drop with no test removed.
- **Workers never run the formatter on `tests/`;** it rewrites unrelated files. A subagent denied a cleanup must report it, and the orchestrator asks Greg instead of running the denied command.
- **Greg playtests from the main checkout,** and Godot rewrites `.import` sidecars there, which blocks `lane.py land`. Commit Godot's sidecar change and land again.
- **Small, test-checkable tickets suit DeepSeek; large UI and networking tickets suit Sonnet.** DeepSeek parked every big polish ticket at least once; Sonnet finished couch play (five tickets) in one lane.
- **Batch landings: one gate for many branches.** Once several reviewed branches wait, make one integration worktree from main, cherry-pick each branch's `main..HEAD` in turn (resolve `progress.md` and `shard_weights` by union), run one full gate, then land it as one branch and close each issue by hand. Nine branches landed behind one 15-minute gate instead of nine in a row. Leave out a branch whose conflict needs real merging (two branches reworking one function); rebase it onto the batch and gate it alone. If the batch gate fails, the failing test names the branch to drop.
- **A script holding a `flock` passes the lock's descriptor to everything it starts.** A main gate or lane started from the landing script kept the lock for its whole run and stalled every landing behind it. Close it on the child: `nohup ... 9>&- &`. `lsof <lockfile>` shows who holds it.
- **Rescue work parked by a loaded gate.** A gate under load dies with exit 124 (shard timeout) or 134 (gdUnit crashes handling a test timeout) after about half the suite, and the lane parks a finished ticket whose own worker gate passed. The park comment names the commit ("committed as `1b11f32`"); the commit is dangling in the lane's clone. Branch it there, fetch it into the landing worktree, cherry-pick, and land with the full gate.
- **Keep every suite under a minute.** A shard is killed at 600 s, so one 3-minute suite plus load timed out whole shards, and workers couldn't run it through `tools/test.sh`'s 120 s limit. Split a slow suite by screen with a shared fixture as soon as it passes a minute.
- **Cap DeepSeek reviews:** add "Answer in under 40 tool calls" to the prompt; an uncapped review hit `--max-turns 100` with no answer. Start a review only after the worktree's conflicts are resolved; `dsreview.sh` reviews whatever HEAD is.
- **Land on the gate's words, not a piped exit code.** `gate | tail -n 2 && merge` merges on `tail`'s exit status; capture the output and grep it for `PASS` before merging.
- **Re-measure a win when its conditions change.** A suite queue beat fixed shards only while shard weights were stale; with fresh weights it lost on both time and CPU, and was reverted.
- **When main goes red, stop queued landings** (by their own PIDs): every full gate behind it fails on the same test and wastes its slot.
- **Red CI is red main, even when every local gate passes.** On 2026-10-21 gridkeep's CI failed for 20 hours unseen: the runner lacked `rg`, and `rg ... && fail` read "command not found" as "no match". After each push to main, check that commit's CI run and hold landings on red; put failed runs in the watcher. A check whose tool is missing or crashes must fail, not pass.

## 5. Keep the queue full

When lanes drain, fan out read-only audits in parallel, each a subagent with one lens: architecture against Greg's rules and the ADRs, tests and coverage, performance (measured, not guessed), agent tooling, and a visual pass on fresh captures. Turn verified findings into tickets and start the next round. After each round, post three bullets in chat for Greg on his phone: what landed, what is red, what needs him. Stop when only `manual` tickets remain, then report: what landed, what each manual ticket needs from Greg, real spend.

Before that report, run the repo's ship steps if its `docs/orchestrating.md` has them (a build and upload for a playtest), on a green final `main`, then send Greg a phone ping with `pushover.py -m "..."` saying what is ready.

A handoff that suggests the next `/swarm` gives the prompt in a code block that starts `/swarm ` (slash, name, space), never in a `>` quote and never as `/swarm.`. This skill has `disable-model-invocation`, so only Greg's typed command starts it; on 2026-10-21 a pasted `/swarm.` went through as plain text and the new session, rightly, refused to run the skill by hand.

## Subagents

- **Models:** pass `model` on every `Agent` call; one without it inherits the orchestrator's Opus. Haiku wherever it can do the job (mechanical work, small well-specified tickets, test-only reviews); Sonnet for judgement, for code, reviews, rebases and audits, as subagents and as loop workers. DeepSeek next, and freely: it is cheap, not a reserve for hard problems. A whole review of a 20-commit refactor cost $0.05 at DeepSeek's prices (the gateway's daily cap counts about twice that), and off-peak hours halve it. Run `deepseek-preflight.py` first: peak (double price) is 01:00–04:00 and 06:00–10:00 UTC on weekdays. Use it for every second-opinion review, for extra lanes (`--harness pi`) when Sonnet lanes are busy or rate-limited, and for bulk reading. Only its daily cap and Greg's `--budget` limit it. Haiku for mechanical work only: status sweeps, running captures and listing results.
- **Every prompt** names one worktree with absolute paths, says what not to touch (the main checkout, other lanes), says no AI attribution lines and no pushing, and ends: "Your FINAL message must be the report itself, under N words". A subagent once returned the word "placeholder"; when a report is empty, inspect its worktree and run the gate yourself.
- **Background:** start subagents and pi-ask in the background, keep orchestrating, and act on each notification.

## Lessons that cost a night to learn

- The DeepSeek balance is the only true cost: read it before and after a run. Each lane's `Real:` line sums that run's own phase costs from DeepSeek's price table; the gateway's counter is shared by every lane, so never add it across lanes (on 2026-10-26 `pi-usage.py` and the old `Real:` lines summed several times the $2.54 the balance actually fell).
- The orchestrator's own context is the biggest Claude cost: on 2026-10-26 it was about 77% of it, grown to 370k and never cleared. `lane.py status` prints it and `session-ctx.py` says OVER past the cap (`SESSION_CTX_CAP`, default 150k); hand off then, even mid-run.

## Handing off past the context cap

Check `lane.py status` at the end of each round and after each landing. When it says OVER, stop taking new work and:

1. Write the handoff (Greg's "Ending a chat" rules in `~/.claude/CLAUDE.md`): lane state, what is parked, what Greg must do. Let it end with the exact first message for the next chat in a fenced code block; `orch-next.py` sends that block, the last one in the file, and refuses a handoff with none. The message starts `/swarm ` and says to resume from the handoff.
2. Run `orch-next.py <handoff> --repo <repo>`. It starts `claude -p` detached with that message and logs to `~/.cache/orch-next/`. The new session has no context: it reads lane state from `lane.py status`, `.git/land/` and the handoff. Check the repo's lanes and watcher survive (they are separate processes) and re-arm the watcher in the new session.
3. Stop. Do not start more lanes or reviews in the old session. `orch-next.py` refuses while an earlier successor still runs.

Claude Code has no hook that starts a session; a scheduled task runs in the cloud, away from the lane clones. So the old session starts its successor itself.
- About 8 lanes saturate a 16-core gate machine. Past that, add no lanes or subagents; `lane.py start` refuses past 10 or a high load.
- Run the monitor through `watch-triage.py`. Every monitor line wakes the orchestrator for a full turn, which costs more than most lanes. Alert only on stops, parks, landings and real failures, never on routine tool failures or lane ticket changes.
- A decision recorded in one session can be reversed in the next. When Greg reverses one, close the ticket as not planned, keep the lane's commits for it off main, and say so in the handoff.

From GregHilston/toolbox, MIT.

More from GregHilston/toolbox