ci-speedup

Files (35)

• SKILL.md
• CODEOWNERS · assets/github/
• action.yml · assets/github/actions/setup/
• merge-loop.json · assets/github/
• agent.json · assets/github/merge-loop/
• environment.json · assets/github/merge-loop/
• loop.sh · assets/github/merge-loop/
• ruleset.json · assets/github/
• agent-task.yml · assets/github/workflows/
• audit-sample.yml · assets/github/workflows/
• deploy.yml · assets/github/workflows/
• main-verify.yml · assets/github/workflows/
• merge-loop.yml · assets/github/workflows/
• pr.yml · assets/github/workflows/
• superfan.yml · assets/github/workflows/
• evals.json · evals/
• 1001-push-success.json · evals/files/runs/
• 1002-push-rerun.json · evals/files/runs/
• 1003-merge-group.json · evals/files/runs/
• 1004-pull-request-cancelled.json · evals/files/runs/
• bots.md · references/
• deploy.md · references/
• flakes.md · references/
• flow.md · references/
• github-build.md · references/
• lanes.md · references/
• ledger.md · references/
• levers.md · references/
• main-health.md · references/
• measuring.md · references/
• merge-loop.md · references/
• merge-path.md · references/
• ci-flow.sh · scripts/
• ci-health.sh · scripts/
• ci-timings.sh · scripts/

SKILL.md (first lines)

# CI Speedup

Find the stage where changes wait between push and deploy, fix that stage with numbers, and write down what changed and when to look again.

- **IS:** measuring where changes wait from push to deploy; a pipeline's critical path; check placement per lane and the required-check set; merge queue sizing or the post-merge design; main-health detection, attribution and revert; the agentic merge loop that reverts, re-lands and repairs without a human; flake policy; deploy throughput; cost per merged change; the ledger.
- **IS NOT:**
  - one PR's red build, conflict or single rerun (`pr-babysitter`)
  - package releases (`autoship`)
  - which checks exist, enforcing a rule, hook wiring, protected paths, or the sandbox rules for an unattended agent job (`codebase-architecture`); this skill owns the merge controller that dispatches those jobs and decides revert, merge and deploy
  - pruning tests (`test-audit`)
  - reviewing a workflow diff (`tidy`)
  - repositories with no pipeline yet (`scaffold-cli`, `scaffold-nextjs`)
  - how many agents to run, or writing the repository's merge-risk policy (this skill reads that policy; it does not author it)

## The Model

The objective is **X = changes merged and deployed per hour**, held inside three budgets and one hard limit:

| Symbol | Budget |
|---|---|
| R | main-red minutes per week |
| C | runner-minutes per merged change: PR runs × pushes, queue runs including discarded speculation, the main run, reruns, and per-push review bots |
| F | p90 time from push to a verdict the author can act on; an agent's context decays while it waits |

Irreversible damage (destructive migrations, infrastructure, leaked secrets) stays at zero whatever it costs X.

Symbols used across every file: λ = arrival rate (merges/hour); T = merge-path CI minutes; p = failure rate on the merged tree, flakes included; p_mb = merge-breakage rate given the head was green; B = batch size; D = speculation depth.

**Finding the constraint.** Pull per-PR timestamps for open, ready, head green, approved, enqueued, merged, main verdict and deployed. Label each stage WAIT or WORK; review and queue time are wait. The constraint is the stage with the most queued minutes per week, and Little's law (WIP = λ × lead time) checks the arithmetic. Exploit it, subordinate the other stages to it, then elevate it, and re-measure after every fix because the constraint moves. Cutting CI duration when CI is not the constraint is a cost lever, not a throughput lever.

**Pipeline duration.** When the pipeline is the constraint, its wall-clock is the slowest chain of dependent jobs, not their sum. A lever that saves runner minutes off that chain lowers C, not F, and the ledger records it as a cost saving.

**Lanes and placement.** Every check runs in one lane: author pre-push, PR gate (exact head), merge path (the merged tree, or nothing), post-merge main (coalesced runs; drives auto-revert), pre-deploy (the exact artifact) or nightly (never a merge verdict). Place each check by asking, in order: irreversible if it ships (block before merge, human tier); merge-breakable, meaning its result can change when merged with other PRs (run it on the merged tree; otherwise at the PR head only); deterministic (only these may drive auto-revert); and what it adds to T against what it catches. The classification table and the YAML are in `references/lanes.md`.

**The default architecture: no merge queue.** PRs merge on evidence for their exact head; main verifies itself after merge in coalesced runs, so each run is a batch; deterministic failures are bisected and reverted by code, and the re-land, conflict and flake repairs go to agents as new tasks; deploys follow the newest verified commit; humans own the high-risk tier, the kill switch and a sampled audit. This is the shape Anthropic runs at 30,000+ merges a day, and it holds at low volume too, where each batch is one commit. Never answer with "require branches up to date" or "require green main before merging": both serialise merges. Flip to a merge queue only when the red budget cannot be met, deploys cannot be limited to verified commits, breakages are mostly nondeterministic, or main cannot be reverted; the flip conditions and the queue math are in `references/merge-path.md`, and the GitHub implementation is in `assets/github/` with `references/github-build.md`.

## Reference Files

| File | Read when |
|------|-----------|
| `references/flow.md` | Step 1, or the user asks where PRs wait: stage definitions, WAIT vs WORK, Little's law, health metrics, cost per merged change, reading `ci-flow.sh` output |
| `references/measuring.md` | Pipeline duration is the constraint, or Step 5 is crediting an after-run: run, job and step timings, the three ratios, the critical path, `merge_group` runs, failure and rerun share, queue time, after-run rules |
| `references/levers.md` | Choosing duration levers: setup, execution, toolchain and image-build levers, what each needs to be true, and the traps |
| `references/lanes.md` | Placing a check; editing `on:`, `if:`, `needs:` or `concurrency:`; changing required checks; a gate that passes while checking nothing |
| `references/merge-path.md` | Whether a flip condition calls for a merge queue, queue sizing, a backed-up queue, ejections, conflicts, merge policy |
| `references/main-health.md` | Main is red, or designing detection, attribution, auto-revert or re-land |
| `references/merge-loop.md` | Designing the agentic merge system: what code decides versus what agents repair, post-merge batching, the agent task contract, the autonomy ladder, loop health, and the agent runtime (Claude in Actions, eve, Devin). Read with `main-health.md` and `bots.md` |
| `references/github-build.md` | Installing or adapting the loop on GitHub: Anthropic's design mapped to rulesets and workflows, setup order, how the controller keeps state, checkout cost, verifying an install |
| `assets/github/` | The templates `github-build.md` installs into `.github/`: `merge-loop.json`, the `loop.sh` controller, the Managed Agents agent and environment, and the `pr`, `main-verify`, `merge-loop`, `agent-task`, `deploy`, `superfan` and `audit-sample` workflows, with the ruleset and CODEOWNERS |
| `references/bots.md` | Building or changing any bot that writes: revert, repair, re-queue, re-land |
| `references/flakes.md` | Intermittent failures, retry or quarantine policy, and before batching or bisecting |
| `references/deploy.md` | Deploy is the constraint, or deploys trail merges |
| `references/ledger.md` | Step 6: the ledger, flow-metric trend, merge-path decision record, instruction-file lines |
| `scripts/ci-flow.sh` | Step 1: per-stage median, p90 and minutes per week for merged PRs with a CONSTRAINT line; `--since`, `--base`, `--workflow`, `--environment`, `--json` |
| `scripts/ci-timings.sh` | Step 2, pipeline: one run's jobs, steps and critical path, or `--runs N` for median, p90, failure, rerun and queue-wait per job; `--event merge_group` for queue runs |
| `scripts/ci-health.sh` | Step 2, main: red episodes on main with first-bad and last-good SHAs, red minutes per week, and flake candidates |

Every script is read-only, takes `--repo owner/name` (or `GH_REPO`), and prints its forms with `--help`. `evals/evals.json` holds scenarios and the with-versus-without comparison for changing this skill; it never loads during a task.

## Workflow

Copy this checklist to track progress:

```text
CI speedup progress:
- [ ] Step 1: Find the constraint (per-stage WAIT and WORK minutes per week)
- [ ] Step 2: Measure the constraint stage (pipeline, merge path, main, or deploy)
- [ ] Step 3: Decide with numbers (predicted X, R, C; one option; its flip condition)
- [ ] Step 4: Change it on a branch; re-time locally; dry-run every bot
- [ ] Step 5: After-measurement from the branch's own runs or a bot replay
- [ ] Step 6: Ledger, decision record, instruction-file lines, re-measure date
```

### Step 1: Find the constraint

Run `scripts/ci-flow.sh` against the default branch and name the constraint in a sentence: which stage, WAIT or WORK, minutes per week, and how it compares with the next stage. A scoped ask (one suite, one job) skips to Step 2's pipeline branch. With fewer than about 20 merged PRs in the window, reason from the workflow config and the volumes the user states, and label every number a projection.

### Step 2: Measure the constraint stage

- **Pipeline:** `scripts/ci-timings.sh --runs 20 --workflow ci.yml` for median and p90 per job, then the single-run form for the steps and the chain. Classify every minute on the critical path as toolchain, gating, setup or execution before proposing anything.
- **Merge path:** `merge_group` runs give T, p and ejections; arrivals from Step 1 give λ.
- **Main:** `scripts/ci-health.sh` gives red episodes, R, and which checks failed on the same SHA they passed on.
- **Deploy:** deployment timings against merge times, and how many merges each deploy carries.

### Step 3: Decide with numbers

State the predicted change in X, R and C, one recommended option, and the measurement that would flip it. When wall-clock falls and runner minutes rise, say by roughly how much and whether the repository pays for minutes.

### Step 4: Change it on a branch

Edit workflow, test and bot configuration in one branch, and re-time locally whatever can be measured without pushing. A change that makes a shard fail is a finding, not a reason to skip the shard. Dry-run every bot that writes by replaying the red episodes from Step 2 and listing what it would have reverted, repaired or re-queued. Building the merge loop: on GitHub, install `assets/github/` by the setup order in `references/github-build.md`, adapt `merge-loop.json` and the setup action to the repository, prove it on a throwaway repository first, ship it at rung 0, and promote only on replay numbers.

Keep the guards. Change detection, concurrency groups, merge checks and deploy gates usually exist because of a past incident; move them, never drop them, and keep the comment that cites the incident next to the guard.

### Step 5: Read the after-measurement

The branch's own runs are the after-measurement; apply the after-run rules in `references/measuring.md` before crediting a lever. When predicted and measured disagree, find out why before adding another lever. A change that waits on a repository setting the user owns gets its after-measurement on the re-measure date, and the ledger says so. Repeat Steps 2 to 5 until the constraint moves to another stage or the remaining options need a decision the user owns.

### Step 6: Ledger and decision record

Write the ledger from `references/ledger.md` into the repository's docs: before, changes, after with run ids, the flow-metric trend row, the decision record with its flip conditions, the queue of options not taken with what each waits on, and the re-measure date. Add the instruction-file lines that stop the layout regressing.

## Permissions and Done

Granted without a check-in, because each is read-only, branch-local or reversible: `gh` reads, running the scripts, local re-timing, pushing the working branch, and opening PRs, revert PRs included.

The user's call, because each spends money or changes what every other change must pass:

- spending: paid runners, higher queue concurrency or speculation depth on a metered plan
- repository settings: rulesets, required checks, enabling the merge queue
- creating the GitHub App, storing its key and the model key, creating the Managed Agents agent and environment, turning on Claude Code Review, and raising `MERGE_LOOP_RUNG` above 0
- merging anything to main

Done means a mergeable change, after-numbers from real runs or a bot replay, a decision record with its flip conditions, and a ledger row with a re-measure date.

## Gotchas

- Sharding a pipeline whose PRs go green in minutes and then wait hours for review or merge changes nothing in X. Measure open to deployed with `ci-flow.sh` before touching the pipeline.
- After a fix the constraint moves: re-timing only the stage just fixed reports a win while X stays flat. Re-run `ci-flow.sh` after every change.

From mblode/agent-skills, MIT.

More from mblode/agent-skills