loop-economist

Files (14)

• SKILL.md
• evals.json · evals/
• BAR.md · evals/gauntlet/
• ROUNDS.md · evals/gauntlet/
• BACKLOG.md · fixtures/thrash-run/docs/
• expect.json · fixtures/thrash-run/
• s1.jsonl · fixtures/thrash-run/transcripts/
• agent-fit.md · references/
• metrics.md · references/
• patterns.md · references/
• verifier.md · references/personas/
• report-template.md · references/
• scorecard.md · references/
• runs.mjs · scripts/

SKILL.md (first lines)

# Loop Economist

`loop-doctor` reads a loop's documents and asks whether the design is
sound. Loop-economist reads what the loop **actually did** — every session
transcript and every commit in a window — and asks a different question:
*what did a unit of shipped change cost, and was it produced by the right
crew?*

A loop can score well on its documents and still spend forty thousand
tokens a tick re-reading a codebase to change one line, or ship and then
un-ship the same file four times. Only the runs show that.

```mermaid
flowchart LR
  A["runs.mjs<br/>transcripts + git · read-only"] --> B["the economics<br/>tokens per commit"]
  B --> C["findings<br/>Leak · Misroute · No-converge · Ill-formed"]
  C --> D["score<br/>6 × 0–5 → verdict"]
  D --> E["prescribe<br/>verifier · routing · subloop · gauntlet"]
  E --> F{"binding constraint?"}
  F -- the actor --> G["review<br/>economics · score · findings · do-next"]
  F -- the input --> H["hand off to<br/>requeue"]
```

## The guard-rail: no overhead on the project

- **Write nothing into the target repo.** The review goes to the
  scratchpad, or to a published page if that is how they read things.
- **Run nothing of the project's.** No install, build or test run. Only
  `git log` and the collector.
- **Never quote a transcript.** Sessions contain the user's own words and
  their code. Cite a session by id and a metric, never by content.
- **Read bounded.** `derived` and `warnings` fully; `sessions` and
  `git.commits` by id only, when a finding needs that one entry's
  evidence. Never read the full `runs.json` — it scales with the window,
  `derived` doesn't.
- **One context, one pass.** No subagents by default — reviewing a
  fan-out problem by fanning out is its own joke.
- **Leave a receipt** in the footer: transcripts read, window, commits,
  repo untouched.

## Workflow

### 1. Collect

```bash
node <skill-dir>/scripts/runs.mjs <repo-path> --since 14d > <scratchpad>/runs.json
jq '{derived, warnings, records, git: {commits: (.git.commits | length), rework: .git.rework}}' <scratchpad>/runs.json
```

`derived` already carries every rate this review reports — tokens per
commit, cache hit rate, subagent share, rework rate, the session-id lists
for thrash and zero-commit — and costs a few hundred tokens regardless of
window. **Read that slice fully; do not read the full `runs.json` top to
bottom.** The `sessions` array and `git.commits` are per-run raw detail
that scales with the window (every session's tool/subagent breakdown,
every commit's numstat) and the review never quotes them — findings cite
a session id or a sha, not their contents. Treat the full file as an
index: pull one entry only when a finding needs to cite that specific
session or commit —

```bash
jq --arg id "<session-id>" '.sessions[] | select(.id == $id)' <scratchpad>/runs.json
```

— never `Read` the whole `runs.json` into context. Sessions (tokens by
kind, models, tools, subagents by type, human turns, interrupts, tool
errors, repeated identical calls, files edited), git (commits, churn,
rework commits), and the loop's records (open/done/vague items) are the
raw material `derived` is computed from; the numbers are the evidence,
not the raw rows.

Pick the window from the loop's cadence: at least **20 ticks or 14 days**,
whichever is longer. Say the window in the first line of the review; every
rate is meaningless without it.

**Write the review from one collection, and quote that collection's numbers.**
A rolling window moves: a review drafted from Monday's run and finished on
Tuesday will state a commit count that no longer exists, and every derived rate
inherits it. Re-run the collector immediately before writing, and verify with
`node evals/bar-check.mjs loop-economist <artifact> --collector <run.json>`,
which compares the headline figures against the collection.

No transcripts (a loop that runs elsewhere, or a fresh machine) → the
budget half is `NOT MEASURED`. Score effectiveness, plan quality and
convergence from git and the records, and say plainly which four numbers
you could not see. Do not infer tokens from diff sizes.

### 2. Read the economics

`references/metrics.md` — each metric with its band and the cause a
reading out of band implies. Establish **tokens per commit** first; it is
the number the reader will remember. Then the three that explain it:
cache hit rate, subagent share, tool calls per edit.

### 3. Find the findings

One row per finding, classified:

- **Leak** — spend that produced no kept change (a zero-commit session, a
  fan-out whose output was re-derived, state re-read every tick).
- **Misroute** — the wrong actor did the work: `agent-fit.md`.
- **No-converge** — thrash, rework, a file the loop cannot leave alone.
- **Ill-formed** — the item had no acceptance test; no actor could have
  converged.

Each row cites a session id, a file or a metric. "Could be more efficient"
is not a row.

### 4. Score

`references/scorecard.md`. Six dimensions 0–5, one-line reason each citing
a number, mean → verdict (`compounding · productive · expensive ·
spinning`), then the caps: nothing shipped caps at *spinning*; more than
three human turns a run caps at *productive*; rework over 0.4 caps at
*expensive*.

**Show the mean and name every cap, including the ones that do not bind.** Add
the numbers up and check the division — a mean that disagrees with its own six
scores is the easiest thing in the review to falsify. Then one line: which caps

From Busy-Office/ai-skills, MIT.

More from Busy-Office/ai-skills