hillclimb

Files (5)

• LICENSE
• SKILL.md
• UPSTREAM.md
• openai.yaml · agents/
• prove-it-works.md · references/

SKILL.md (first lines)

# Hillclimb

**You own the metric and the experiment's integrity. Supervise and review. Delegate the attempts.** For sustained, iterative improvement of one measurable thing against a target. A one-off fix needs no loop. This is the loop.

Core discipline: one change, one measurement, keep or revert. Never stack untested changes, and never claim a win from code inspection. Before the first keep, read `references/prove-it-works.md` in this skill's folder.

1. Ground the workload and architecture before choosing the metric. Read the code that produces the metric end to end, name the realistic workload dimensions that can move the result (data size, history, state, concurrency), and select a case that reproduces the user's complaint. If no case reproduces it, fix the repro instead of hillclimbing. Then fix one metric, the direction that counts as better, and a checkable stop predicate that pairs a target with a floor on attempts so a lucky early win can't end the run (the example "at least 50% better than baseline and at least 10 iterations" is this shape). Use the user's numbers when given, otherwise agree them.
   If the metric is a score on labeled or sampled data, split the data before the first measurement: a dev set for the loop, and a held-out test set that no attempt, hypothesis, or subagent sees. The loop measures only the dev set.
2. Build the measurement harness, prove its sensitivity, then freeze it. Run contrasting realistic workloads and confirm the target case reproduces the symptom while easier cases separate as expected. If the harness cannot distinguish them, revise the workload or metric. Before you freeze the harness, call the Skill tool with `benchmark-checklist` to vet it, and make it print its error count and a count of the work done. Once frozen, one repeatable command emits the metric, sampled enough to clear the noise (median of N, not a single run). Record the baseline metric and a green run of the regression gate (the tests that must keep passing) before any change.
3. Open a decision log: a `decision.tsv`, one row per attempt: id, hypothesis, change, before, after, delta, tests, verdict (kept or reverted), note. Read it before each attempt. Keep it out of the tree (gitignored).
4. Ground each hypothesis in the architecture model from step 1, so it names a specific mechanism ("defer X off the boot path because it blocks first paint"), not "try memoizing something". For a perf metric, order hypotheses cheapest first: don't do it, do it once, do it less, do it later, do it when nobody waits, do it concurrently, do it cheaper.
5. Loop, one hypothesis per iteration:
   - Hand the change to a subagent with a fresh context window and a tight scope. Supervise and review the diff rather than typing it, so your context holds the loop, not the code. When several independent hypotheses are live, fan them to parallel subagents, each in its own worktree, and measure each one alone: parallel runs on one machine measure each other's load.
   - Measure before and after with the frozen harness, and run the regression gate.
   - Accept only when the metric moves past noise and the gate stays green. Otherwise revert the change in full. A tweak that "might help" is not kept.
   - One commit per accepted fix, staging only the files you changed (`git add <files>`, never `-A`). Log the row either way, kept or reverted.
   Each iteration ends in a check before the next begins. If the run is unattended, wake on the event you wait for (a background job that ends, a watched file), with a long time-based heartbeat as fallback.
6. Push past the first plateau. On a stall, several rejects in a row, pivot category, combine near-misses, re-read the source, or try something more radical before concluding the hill is climbed. Correctness and simplicity outrank the number. Revert a win that breaks behavior, and keep a simplification that holds the number.
7. Stop when the predicate is met, or when the remaining ideas are marginal and not worth their cost. Don't relax the predicate to meet it, and don't quit while cheap untried hypotheses remain. If you are stuck, surface it instead of spinning.
   With a held-out test set, score the baseline and the final version on it once, after the loop stops. A test score well below the dev score means the loop fit the dev set: report it, and never tune against the test set.
8. Ship the accepted commits in the order they landed, the way the repo ships: a push or a PR.

**Reply:** the metric and target, baseline to final with the percent delta, the held-out test score for both if there is a test set, iterations run (kept vs reverted), each accepted fix on one line, the `decision.tsv` path, and the best idea you would try next if pushed further.

From VanTanev/skills, MIT.

More from VanTanev/skills