autoresearch

Files (2)

• SKILL.md
• autoresearch_helper.py

SKILL.md (first lines)

# autoresearch

Systematically improve one measurable metric — training loss, test runtime, bundle
size, build time, render latency — through an autonomous experiment loop. Core
discipline: **one change, one measurement, keep or revert. The data decides**
([prove-it-works](../prove-it-works/SKILL.md)); never claim a win from code inspection,
and never stack untested changes. All state lives in files so a fresh session with no
memory resumes exactly where the last one stopped.

## Setup

### 1. Ground the experiment before choosing the ruler

Read the source files deeply; understand the workload before writing anything. Name the
realistic workload dimensions that can move the result (data size, history, state,
concurrency) and select a case that reproduces the user's actual complaint — if no case
reproduces it, fix the repro instead of optimizing. Then fix, with the user or from
their numbers:

- **Goal** and **primary metric**: name, unit, direction (e.g. `val_bpb`, lower is
  better). Secondary metrics are tradeoff monitors, not targets.
- **Command**: what runs the workload.
- **Files in scope** and **off-limits**; **constraints** (tests green, no new deps).
- **Stop predicate**: a checkable target *paired with a floor on attempts* so a lucky
  early win can't end the run ("at least 20% better than baseline AND at least 10
  experiments"), or a time/count budget, or run-until-interrupted. Never relax the
  predicate to declare victory.

### 2. Build and freeze the harness

Work on a dedicated branch: `git checkout -b autoresearch/<goal>-<date>`.

Write `autoresearch.sh` (`set -euo pipefail`): a fast pre-check (syntax in <1s), then
the workload, then structured `METRIC name=value` lines to stdout. Fast noisy
benchmarks (<5s) run the workload several times inside the script and report the
median; slow ones don't need it.

**Prove the ruler's sensitivity, then freeze it** ([build-the-lever](../build-the-lever/SKILL.md)):
run contrasting workloads and confirm the target case reproduces the symptom while
easier cases separate as expected. A ruler that can't distinguish them gets revised
before any experiment. Once frozen, changing the script invalidates every earlier
number — treat that as a restart, not a tweak.

If constraints require correctness, write `autoresearch.checks.sh` (tests, typecheck)
— the regression gate. Record a green gate run *before* any change.

Write `autoresearch.md`, the living research document a fresh agent can run the loop
from: objective and workload, metrics, how to run, files in scope, off-limits,
constraints, stop predicate, and a "What's Been Tried" section that accumulates wins,
dead ends, and insights.

Initialize the log and commit:

```bash
python3 autoresearch_helper.py init --jsonl autoresearch.jsonl --name '<goal>' \
  --metric-name '<metric>' --direction <lower|higher>
git add autoresearch.md autoresearch.sh autoresearch.jsonl && git commit -m "autoresearch: init"
```

Run the baseline (`bash autoresearch.sh`), log it as experiment #1 with
`status keep --description "baseline"`.

## The loop

**Loop until the stop predicate is met.** Never ask "should I continue?" — proceed and
present; the checkpoint files are the supervision surface. Stop early only for the
user's interrupt or exhausted context (see Resume).

Per experiment:

1. **Choose a hypothesis.** Read `autoresearch.md` ("What's Been Tried") and
   `autoresearch.ideas.md` first so the search accumulates instead of circling. A
   hypothesis names a specific mechanism ("defer X off the boot path because it blocks
   first paint"), never "try memoizing something". The best ideas come from re-reading
   the source and the data, not random variation.
2. **Make the change.** One hypothesis per experiment, files in scope only.
3. **Measure.** `timeout 600 bash autoresearch.sh`; parse the METRIC lines. Crash or
   timeout: log as `crash`, revert. Then the gate: `timeout 300 bash
   autoresearch.checks.sh` if it exists; failure logs `checks_failed` and reverts.
4. **Evaluate** against the current best:
   `python3 autoresearch_helper.py evaluate --jsonl autoresearch.jsonl --metric <v> --direction <d>`.
   Keep when the primary metric improves past noise AND the gate is green; also keep
   simpler code at equal performance (deleting code for the same number is a win —
   [lazy-ladder](../lazy-ladder/SKILL.md)). Everything else reverts **in full**: a
   tweak that "might help" does not ride along. Ugly complexity for a tiny gain is a
   discard — correctness and simplicity outrank the number.
5. **Record.** Log to JSONL *before* commit/revert so the entry survives either way,
   always with ASI (`{"hypothesis": ...}`, plus `rollback_reason` and
   `next_action_hint` on failures — ASI is the only structured memory that survives
   reverts). On keep: `git add -A && git commit` with the result in the message — one
   commit per kept fix. On revert: back up the state files
   (`autoresearch.{jsonl,md,ideas.md}`), `git checkout -- . && git clean -fd`, restore
   them.
6. **Journal.** Every few experiments, update "What's Been Tried"; append deferred
   ideas to `autoresearch.ideas.md`; prune stale entries.

**Plateau means pivot, not stop.** Several rejects in a row: change hypothesis
category, combine near-misses, re-read the source, or try something structurally more
radical before concluding the hill is climbed. Repeatedly reverting the same idea is
thrash — go structurally different. Confidence score (MAD-based, from the helper) is
advisory: below 1.0× noise, re-run the same experiment to confirm before keeping.

## Resume and context

All state is in the files. After ~15 experiments in one session, context is heavy:
update `autoresearch.md`, commit state files, stop. A fresh session resumes by reading
`autoresearch.md`, `python3 autoresearch_helper.py status --jsonl autoresearch.jsonl`,
and `git log --oneline -20`. Running under an orchestrator: the bd ticket carries the
goal, predicate, scope, and constraints; checkpoint experiment counts on the ticket.

## Finalization

When the predicate is met (or the run ends):

1. `python3 autoresearch_helper.py summary --jsonl autoresearch.jsonl` and review
   `git log --oneline --stat <merge-base>..HEAD`.
2. Group kept experiments into logical changesets that share no files, so each branch
   merges independently; present the grouping (skip the wait when running as a worker).
3. Per group: branch from the merge-base, check out the group's files from the session

From baranbingol1/bbstack, MIT.

More from baranbingol1/bbstack