test-prune

Files (26)

• SKILL.md
• prune.py
• prune_repo.py
• prune_review.py
• prune_runs.py
• prune_source.py
• prune_strengthen.py
• conftest.py · tests/
• prune_harness.py · tests/
• test_prune_guards.py · tests/
• test_prune_inputs.py · tests/
• test_prune_review.py · tests/
• test_prune_source.py · tests/
• test_prune_strengthen.py · tests/
• test_prune_unattended.py · tests/
• gh · tests/testdata/
• .gitignore · tests/testdata/repo/
• Makefile · tests/testdata/repo/
• mutation_baseline.json · tests/testdata/repo/backend/
• concentration.py · tests/testdata/repo/backend/services/
• rebalancer.py · tests/testdata/repo/backend/services/
• conftest.py · tests/testdata/repo/backend/tests/
• test_concentration.py · tests/testdata/repo/backend/tests/
• test_rebalancer.py · tests/testdata/repo/backend/tests/
• fake_engine.py · tests/testdata/repo/
• kill_map.json · tests/testdata/repo/

SKILL.md (first lines)

# test-prune: delete what the mutation ratchet proves redundant, strengthen what it proves weak

You are the orchestrator. `prune.py` owns every git, make and file operation and every safety
guard; the subagents only write JSON files or tests. Each role is a fresh subagent with no
context from the others: the reviewer reads the fixer's proposals, reasons included, and judges
them against the code itself.

**Publishing belongs to prune.py.** In both modes you never run `git push`, `git merge`,
`gh pr create` or `gh pr merge` yourself, and neither does any subagent; `prune` and
`strengthen-finish` push their own `chore/test-*` branch and open its PR, and nothing merges.

The target repo provides the make targets `test`, `mutation`, `mutation-check`,
`mutation-baseline`, `test-candidates`, `prune-gate` and `crap`, and the layout `backend/tests/`,
`backend/mutation_baseline.json`. Every command:

```bash
python3 "<skill-dir>/prune.py" <command> [--state DIR] [--date YYYY-MM-DD]
```

Run it from inside the target repo. State lives in `<repo root>/.test-prune/`, which prune.py keeps
out of git through `info/exclude`; each command reads what the previous one wrote there. Subagents
write their JSON files there too. Never put it under `.git/`: Claude Code refuses agent writes there. Exit 0 is done (including nothing to
do), 1 is a failed make target, gate or guard, 2 is bad input or an unmet precondition; the
message names the offending ids or files. On exit 2 fix the input and rerun the same command; on
exit 1 stop and report the output.

Every command records itself in `<state>/status.json`. A CI job reads it after the headless
session (which cannot set the job's exit code) and fails the job when its `exit` is not 0; an
unattended run stays red there until the strengthen loop ends (`nothing to strengthen`, or
`strengthen-finish` exit 0). End a headless session by reporting its `reason` as the final line.
`--state` must be empty, new, or an earlier run's state dir.

When a push succeeded but `gh pr create` failed (exit 1, the branch is on origin with no PR),
open the PR by hand from `<state>/prune-pr.md` or `<state>/strengthen-pr.md` after fixing `gh`;
rerunning the command refuses because the branch exists.

**A run needs a checkout dedicated to it** for its whole duration: the CI checkout, or a worktree.
If anything else edits a file there mid-run, the run stops with exit 1 rather than overwrite it,
leaves the checkout as it is (HEAD may stay detached), names the files, and nothing is lost; keep
or discard the change, then check out the base branch. Races beyond that check are out of scope.

**Runtime.** Each cold `make mutation` takes about 30 minutes on a real repo, and a run makes
several (collect, each safety-check run, each strengthen proof). Run long commands with a timeout of
an hour or more, or in the background, and wait for them.

## Modes

- **Interactive** (default): nothing in the working tree changes before the human approves the
  batch. Stop after `batch`, show its output, and run `apply` only on an explicit yes.
- **Unattended** (`collect --unattended`, run headless by the weekly CI job): only protected
  candidates, at most 10 deleted test ids per run (a parametrized test goes whole, every case
  counted; one with more than 10 cases is skipped, the rest left for next week), one PR from
  `chore/test-prune-<date>` and one from `chore/test-strengthen-<date>`, both from the base
  commit. The base branch never moves. A merge is the human's approval, so the run ends with open
  PRs.

Two kinds of accepted proposal reach a human only: equivalent mutants (the skill writes no
equivalent id into the baseline) and "strengthen this candidate with this assertion" verdicts
(the skill applies no assertion to an existing candidate). `batch` prints both; unattended, every
PR body and the job summary carry them.

## Prune loop

1. `collect [--unattended | --protected-only]`. Needs a clean tracked tree on a branch. Runs the
   base checks, a cold mutation run and `make test-candidates`, then writes
   `fixer-input.json`. When it prints `no candidates` / `no protected candidates`, the weak spots
   may still need the fixer; continue.
2. **Fixer.** Dispatch a subagent with `<state>/fixer-input.json`, read access to the repo, and
   this brief: for every candidate decide `keep`, `delete` into one named test that contains it,
   or `strengthen` with one named assertion; for every weak-spot group decide `strengthen`
   (which test to add to or strengthen) or `equivalent` (with the argument why no test can tell
   the mutant apart). A group with both kinds of mutant takes two entries, one `strengthen` and
   one `equivalent`, each listing its own disjoint `mutant_ids`. Compare assertions, not coverage: a candidate whose assertions pin a value,
   boundary or error the container never checks is `keep` or `strengthen`. It writes:

   ```json
   {"candidates": [{"test_id": "...", "verdict": "keep|delete|strengthen",
                    "into": "<test id, delete only>", "assertion": "<strengthen only>", "reason": "..."}],
    "weak_spots": [{"group": "...", "verdict": "strengthen|equivalent",
                    "mutant_ids": ["<subset of the group's ids, omit for all>"],
                    "test": "<test id or test file>", "reason": "..."}]}
   ```

   One entry per candidate, one per group (or the strengthen + equivalent pair), no others.
3. `review-input --proposals <fixer file>` validates the file and writes
   `<state>/review-input.json`: each proposal with the candidate's source and assertions and,
   for a deletion, the containing test's source and assertions. A proposal's `id` is the
   candidate's test id or the group; a split group's two proposals are `<group>:strengthen` and
   `<group>:equivalent`.
4. **Reviewer.** Dispatch a *different* fresh subagent with `<state>/review-input.json` and read
   access to the repo. Brief: accept a deletion only when every assertion of the candidate is
   implied by an assertion of the `into` test (same inputs or a superset, an equal or stricter
   check); covering the same lines is not enough. Accept a strengthen proposal only when the
   named assertion checks behaviour a user would notice. Accept an equivalent only with a
   concrete argument that no input distinguishes the mutant. Every reason names the assertion or
   input it rests on. It writes, one entry per proposal:

   ```json
   {"reviews": [{"id": "<proposal id>", "verdict": "accept|reject", "reason": "..."}]}
   ```

5. `batch --reviews <reviewer file>` writes `batch.json` and prints the accepted deletions (each
   with its containing test and the reviewer's reason), strengthen items, equivalent proposals,
   skipped deletions with the reason, and rejections. It edits nothing.
   - Interactive: show that output to the human and stop. After approval, `apply` applies the
     deletions, runs the safety check, restores every culprit and commits on the current branch.
   - Unattended: `prune` applies the batch, runs the safety check, `make test`, commits, runs
     `make prune-gate` and, only when it passes, pushes the branch and opens the PR.

**Safety check** (inside `apply` and `prune`): a cold mutation run plus `make mutation-check`
on the batch; every deletion that lets a module exceed its baseline is restored and reported.

## Strengthen loop

Runs after the prune loop, also in unattended mode, independently of the prune PR: run it even
when `prune` exited 1 (the status keeps that failure, so the job still goes red).

1. `strengthen-start` writes `<state>/strengthen-input.json` (each accepted weak-spot strengthen
   item with the mutant ids it must kill) and prints the commit to build on; unattended it

From jesdi/general-skills, MIT.

More from jesdi/general-skills