experiment-plan

Files (1)

• SKILL.md

SKILL.md (first lines)

# experiment-plan

**Trigger**: User has a refined idea or research question and asks what experiments to run, or auto-invoked after `idea` (refine mode). For hyperparameter sweeps over an already-planned experiment use `experiment-dse`; for launching a single config use `experiment-run`; for analyzing results use `experiment-analyze`.

**Process**:

### 1. Read context

- Refined idea at `methodology/ideas/{slug}.md` (required — if missing, route to `idea` refine mode first)
- Existing experiment configs at `experiments/` (reuse schema conventions; don't invent a new config shape mid-project)
- `resources/papers/` notes for prior baselines + their reported numbers (you will need these for the baselines table)
- `config.yaml` resource budgets (GPU budget, deadline)

If multiple refined ideas exist, ask the user which to plan. Do not combine two ideas into one plan — ablation logic gets tangled fast.

### 2. Extract the testable claims

From the refined idea, list the claims the paper will make in the order they'd appear in a Results section. Each claim becomes one row. Keep claims **falsifiable**:

| # | Claim | How experiment tests it |
|---|---|---|
| C1 | Method M outperforms baseline B on task T | Direct comparison M vs B on T, N seeds |
| C2 | The gain attributes to component X | Ablation: M-without-X vs M |
| C3 | Gain persists at scale (parameter count ≥ S) | Scaling experiment: M and B at 3 sizes |
| C4 | Gain generalizes across domains | M vs B on held-out domains |

Rules:
- Every claim must have exactly one primary experiment. If a claim needs two, it's really two claims — split it.
- Mark claims as **core** (must land for the paper to exist) or **supporting** (strengthens narrative but removable if budget tightens).
- If a claim has no feasible experiment, flag it and refine — unfalsifiable claims shouldn't go into a plan.

### 3. Variable mapping

For each experiment row in the claims table, spec the variables:

- **Independent (what we change)**: method, hyperparameters, data, seed, scale — exactly what is varied, with the specific set of values.
- **Dependent (what we measure)**: primary metric (with its exact definition — "BLEU-4 on WMT14 en-de test set" not just "BLEU"), secondary metrics, and failure indicators (NaN loss rate, OOM frequency).
- **Controlled (held constant)**: hardware, library versions, data preprocessing, eval harness version. Pin these so a future rerun reproduces.
- **Nuisance (expected to vary, averaged out)**: random seed (N ≥ 3 unless compute-bound; N = 5 ideal), data shuffling order, minor non-determinism.

Output a variables table per experiment. Every hyperparameter present in any config must appear in exactly one of the 4 columns — nothing lives outside the table.

### 4. Baselines + fair-comparison checks

For each experiment, pick baselines with a written justification:

| Baseline | Version / config | Why this baseline | Reported number (if any) | Our reproduction target |
|---|---|---|---|---|
| B1 | {exact ref} | {why fair} | {from paper} | {±tolerance we accept} |

Fair-comparison rules:
- Same data split, same eval harness, same metric definition. If any differs, note explicitly why and how you'll reconcile.
- Baselines run under the same controlled variables as the method. No "our method got X tokens; baseline got 0.5X" without correcting.
- If a baseline's reported number can't be reproduced within tolerance, budget time to either reproduce it or document the discrepancy before claiming a win over it.

Include **at least one trivial baseline** (random / majority-class / frozen-pretrained) as a sanity floor.

### 5. Ablation schedule

For claim C2 (and any "the gain attributes to X" claims):

| Ablation | Removes / replaces | Predicts | Budget |
|---|---|---|---|
| A1: no-X | Method M with component X disabled | metric drops to within baseline ± Δ | 1× full-run cost |
| A2: X-only | Only X, rest replaced with baseline defaults | metric improves over baseline by ≥ Δ' | 1× full-run cost |

Ablations come in **pairs** when possible (remove X → should hurt; add X alone → should help). One-directional ablations are weaker evidence.

Order ablations by cost ascending, expected-signal descending. If budget is tight, run the cheap, high-signal ablations first.

### 6. Resource estimate + schedule

Produce a compact table:

| Row | Configs | Seeds | Cost per config | Total GPU-h | Wall-clock (parallel) |
|---|---|---|---|---|---|
| Main comparison | 2 methods × 2 data regimes | 3 | 8 h | 96 | 2 days |
| Ablations | 4 | 3 | 8 h | 96 | 2 days |
| Scaling | 3 sizes × 2 methods | 3 | varies | 180 | 4 days |
| **Total** | | | | **372 GPU-h** | **~8 days** |

- Compare total against `config.yaml` budget + deadline. If over, flag which ablations / seeds to cut first (mark them "trimmable" in the claims table, not "core").
- Reserve 20% headroom for reruns / debugging / surprise OOMs. A plan that assumes zero reruns will blow its budget.

### 7. Save the plan

Write to `experiments/{exp_name}/plan.md`:

```markdown
---
exp_name: "{exp_name}"
idea_ref: "methodology/ideas/{slug}.md"
status: "planned"
created: "YYYY-MM-DD"
deadline: "YYYY-MM-DD"
budget_gpu_h: {N}
seeds: {N}
---

## Claims Table
{from step 2}

## Variable Mapping (per experiment)
{from step 3}

## Baselines
{from step 4}

## Ablation Schedule
{from step 5}

## Resource Estimate
{from step 6}

## Execution Order
1. {trivial baselines — fail fast if anything is broken}
2. {core claim main comparison — fail the paper fast if the headline doesn't land}
3. {ablations — only if main comparison lands}
4. {scaling / generalization — strengthen the narrative}

From Shiien/Self-Evolved-Research-Framework, MIT.

More from Shiien/Self-Evolved-Research-Framework