autoresearch

Files (6)

• SKILL.md
• openai.yaml · agents/
• setup-template.md · assets/
• evals.json · evals/
• trigger-queries.json · evals/
• experiment-guide.md · references/

SKILL.md (first lines)

# Autoresearch

Use this skill when the goal is to improve a specific, measurable outcome through repeated automated experiments — write a change, measure it, keep it or revert it, repeat. You define the metric; the agent runs the loop autonomously until interrupted or the budget is exhausted.

This is distinct from test-driven development (writing tests for behavior) and systematic debugging (diagnosing a known failure). Use this skill when optimization through iteration is the work.

## Use this skill when

- You want to iteratively improve a measurable outcome: execution time, bundle size, test pass rate, build time, latency, coverage, complexity, or a custom benchmark score.
- You have a metric command that produces a numeric result and you want the agent to run experiments autonomously until interrupted or a budget is reached.
- The right solution is not obvious upfront and you want to explore the space empirically.
- You want to hill-climb: keep changes that help, discard changes that do not, and log everything.

## Do not use this skill when

- The task is writing tests for a new feature or behavior (use `test-driven-development`).
- The goal is diagnosing why something is broken (use `systematic-debugging`).
- There is no measurable metric — the outcome is purely qualitative or subjective.
- The codebase does not have a git repository or the user has not confirmed a dedicated experiment branch is acceptable.

## Routing boundary

| Situation | Use this skill? | Route instead |
| --- | --- | --- |
| Writing failing tests before implementing a feature | No | `test-driven-development` |
| Diagnosing why a test or build is failing | No | `systematic-debugging` |
| Iteratively optimizing a benchmark or metric | Yes | — |
| Improving code quality with no numeric signal | No | `systematic-debugging` or `verification-before-completion` |
| Running a single targeted experiment to validate a hypothesis | No | Implement directly; this skill is for multi-experiment autonomous loops |

## Inputs to gather

**Required before editing**

- **Goal**: what you are trying to improve (e.g., execution time, pass rate, bundle size).
- **Metric command**: the exact shell command that produces the metric (e.g., `npm run benchmark`, `go test -bench=. ./...`, `hyperfine './build.sh'`).
- **Metric extraction**: how to pull the numeric value from the output (regex, JSON field, specific line).
- **Direction**: `lower_is_better` or `higher_is_better`.
- **In-scope files/directories**: what the agent is allowed to edit.
- **Out-of-scope files/directories**: what must not be touched.

**Helpful if present**

- **Max experiments**: a count or `unlimited` (default: unlimited, stop only on interrupt).
- **Constraints**: time budget per run, dependency policy, test-passing requirement, API compatibility, memory limits.
- **Simplicity policy**: default is "simpler is better — weigh complexity cost against improvement magnitude."

**Only investigate if encountered**

- Existing benchmark history or prior result logs.
- CI-enforced test gates that must stay green throughout the loop.

## First move

1. Confirm all required inputs are known. If the metric command or extraction method is unclear, ask before creating the branch.
2. Create a dedicated experiment branch: `git checkout -b autoresearch/<tag>` (use today's date as the tag, e.g., `autoresearch/2025-07-09`).
3. Run the metric command on the unmodified code and record the baseline value before any experiments begin.

## Workflow

### Phase 1: Setup confirmation

Summarize all parameters in a table before proceeding:

| Parameter | Value |
| --- | --- |
| Goal | … |
| Metric command | … |
| Metric extraction | … |
| Direction | lower is better / higher is better |
| In-scope files | … |
| Out-of-scope files | … |
| Constraints | … |
| Max experiments | … |
| Simplicity policy | … |

Do not start the loop until confirmed.

### Phase 2: Branch and baseline

1. Create branch: `git checkout -b autoresearch/<tag>`.
2. Read all in-scope files to build full context of the current state.
3. Initialize `results.tsv` in the repo root with the header row (see `references/experiment-guide.md` for the TSV format). Add `results.tsv` and `run.log` to `.git/info/exclude` so they stay untracked without modifying any tracked file.
4. Run the metric command on unmodified code. Record as experiment `0` with status `baseline`.
5. Report baseline to the user before starting the loop.

### Phase 3: Experiment loop

Run until `MAX_EXPERIMENTS` is reached or the user interrupts. Do not stop to ask permission between iterations.

For each experiment:

1. **Think** — analyze previous results and current code. Generate a hypothesis. Consult `references/experiment-guide.md` for strategy ordering (low-hanging fruit first, diversify after plateaus, combine winners, etc.).
2. **Edit** — modify only in-scope files. Keep changes focused and minimal per experiment.
3. **Commit** — `git add <changed files> && git commit -m "experiment: <short description>"`.
4. **Run** — execute the metric command. Redirect to `run.log` (`<command> > run.log 2>&1`).
5. **Measure** — extract the metric from `run.log`. If extraction fails, read the last 50 lines for the error.
6. **Decide**:
   - **Improved**: keep the commit. Update the running best.
   - **Same or worse**: revert with a revert commit — `git revert HEAD --no-edit`. Log status `discard`.
   - **Crash/error**: if a quick single-line fix is obvious, apply it in a new commit and re-run once. If still broken after one fix attempt, revert with `git revert HEAD --no-edit` (or revert both commits if two were made) and log status `crash`. Do not use `--amend` or `reset --hard`.
7. **Log** — append one row to `results.tsv`: `experiment_number  commit_hash  metric_value  status  description`.
8. **Continue** — go to step 1.

### Phase 4: Report

When the loop ends:

1. Print the full `results.tsv` as a formatted table.
2. Summarize: total experiments, kept / discarded / crashed counts, baseline vs. final metric, improvement percentage, top 3 most impactful changes.
3. Show the kept-commit log: `git log --oneline <start_commit>..HEAD`.
4. Recommend next steps: ideas that were too risky or complex for the automated loop.

## Outputs

- `autoresearch/<tag>` branch containing only kept (improving) experiment commits.
- `results.tsv` (untracked) — full experiment journal.
- `run.log` (untracked) — output from the most recent metric run.
- A written Phase 4 summary report in the chat.

From matt-riley/agent-skills, GPL-3.0.

More from matt-riley/agent-skills