autoresearch

Files (1)

• SKILL.md

SKILL.md (first lines)

# Autoresearch: Autonomous Iterative Experimentation

An autonomous experimentation loop for any programming task. You define the goal and how to measure it; the agent iterates autonomously -- modifying code, running experiments, measuring results, and keeping or discarding changes -- until interrupted.

This skill is inspired by [Karpathy's autoresearch](https://github.com/karpathy/autoresearch), generalized from ML training to **any programming task with a measurable outcome**.

## Agent Behavior Rules

1. **DO** guide the user through the Setup phase interactively before starting the loop.
2. **DO** establish a baseline measurement before making any changes.
3. **DO** commit every experiment attempt before running it (so it can be reverted cleanly).
4. **DO** keep a results log (TSV) tracking every experiment.
5. **DO** revert changes that do not improve the metric (git reset to last known good).
6. **DO** run autonomously once the loop starts -- never pause to ask "should I continue?".
7. **DO NOT** modify files the user marked as out-of-scope.
8. **DO NOT** skip the measurement step -- every experiment must be measured.
9. **DO NOT** keep changes that regress the metric unless the user explicitly allowed trade-offs.
10. **DO NOT** install new dependencies or make environment changes unless the user approved it.

## Phase 1: Setup (Interactive)

Before any experimentation begins, work with the user to establish these parameters.
Ask the user directly for each item. Do not assume or skip any.

### 1.1 Define the Goal

Ask the user:

> **What are you trying to improve or optimize?**
>
> Examples: execution time, memory usage, binary size, test pass rate, code coverage,
> API response latency, throughput, error rate, benchmark score, build time, bundle size,
> lines of code, cyclomatic complexity, etc.

Record the user's answer as the **goal**.

### 1.2 Define the Metric

Ask the user:

> **How do we measure success? What exact command produces the metric?**
>
> I need:
>
> 1. **The command** to run (e.g., `dotnet test`, `npm run benchmark`, `time ./build.sh`, `pytest --tb=short`)
> 2. **How to extract the metric** from the output (e.g., a regex pattern, a specific line, a JSON field)
> 3. **Direction**: Is lower better or higher better?
>
> Example: "Run `dotnet test --logger trx`, count passing tests. Higher is better."
> Example: "Run `hyperfine './my-program'`, extract mean time. Lower is better."

Record:

- `METRIC_COMMAND`: the command to run
- `METRIC_EXTRACTION`: how to extract the numeric metric from output
- `METRIC_DIRECTION`: `lower_is_better` or `higher_is_better`

### 1.3 Define the Scope

Ask the user:

> **Which files or directories am I allowed to modify?**
>
> And which files are OFF LIMITS (read-only)?

Record:

- `IN_SCOPE_FILES`: files/dirs the agent may edit
- `OUT_OF_SCOPE_FILES`: files/dirs that must not be modified

### 1.4 Define Constraints

Ask the user:

> **Are there any constraints I should respect?**
>
> Examples:
>
> - Time budget per experiment (e.g., "each run should take < 2 minutes")
> - No new dependencies
> - Must keep all existing tests passing
> - Must not change the public API
> - Must maintain backward compatibility
> - VRAM/memory limit
> - Code complexity limits (prefer simpler solutions)

Record as `CONSTRAINTS`.

### 1.5 Define the Experiment Budget (Optional)

Ask the user:

> **How many experiments should I run, or should I just keep going until you stop me?**
>
> You can say a number (e.g., "try 20 experiments") or "unlimited" (I'll run until you interrupt).

Record as `MAX_EXPERIMENTS` (number or `unlimited`).

### 1.6 Simplicity Criterion

Inform the user of the default simplicity policy:

> **Simplicity policy (default):** All else being equal, simpler is better. A small improvement
> that adds ugly complexity is not worth it. Removing code while maintaining or improving
> the metric is a great outcome. I'll weigh the complexity cost against the improvement
> magnitude. Does this policy work for you, or do you want to adjust it?

Record any adjustments as `SIMPLICITY_POLICY`.

### 1.7 Confirm Setup

Summarize all parameters back to the user in a clear table:

| Parameter          | Value                        |
| ------------------ | ---------------------------- |
| Goal               | ...                          |
| Metric command     | ...                          |
| Metric extraction  | ...                          |
| Direction          | lower is better / higher ... |

From thimslugga/agent-plugins, MIT.

More from thimslugga/agent-plugins