experiment-dse

Files (1)

• SKILL.md

SKILL.md (first lines)

# experiment-dse

**Trigger**: User asks to sweep, tune, or explore a hyperparameter space, typically with an existing `experiment-plan` or a set of parameters + ranges. For launching a single committed config use `experiment-run`; for designing the overall experiment spec first use `experiment-plan`; for post-sweep analysis of the winning config's final numbers use `experiment-analyze`.

**Process**:

### 1. Define the search space

- If `experiments/{exp_name}/plan.md` exists, pull the variables table — parameters flagged with ranges (not single values) are the DSE axes.
- Otherwise, ask the user to enumerate each parameter as one of:
  - **Continuous** with range + sampling prior (`lr ∈ [1e-5, 1e-2]`, log-uniform)
  - **Discrete ordinal** (`batch_size ∈ {16, 32, 64, 128}`)
  - **Categorical** (`optimizer ∈ {AdamW, SGD+M, Lion}`)
- **Objective function**: exactly one primary metric to optimize + direction (max / min). Secondaries are reported but do not drive search.
- **Constraints**:
  - Hard budget (max GPU-hours total / max wall-clock)
  - Hard per-config limits (memory ceiling, per-run wall-clock for early-stop)
  - Soft thresholds (minimum acceptable primary metric — below this, don't bother running secondaries)

If the search space is too large for the budget, trim **before** any runs — either reduce ranges, drop parameters to ablation-only, or promote the search to `experiment-plan` if structural changes are needed. Do not start a sweep that can only cover 5% of the space — the result will be misleading.

### 2. Choose search strategy

Pick by space size and prior knowledge, not reflex. Grid search is the default lazy choice and is almost always wrong at scale.

| Strategy | Fits when | Budget needed |
|---|---|---|
| **Grid** | ≤ 50 configs total, no prior on which axes matter | #configs = ∏ axis_sizes |
| **Random** | > 50 configs, high-dim space, unknown sensitivity | 30–100 configs typical |
| **Bayesian-inspired sequential** | Expensive per-config (≥ 4 GPU-h), smooth objective | 20–60 configs, iterative |
| **Latin hypercube / Sobol** | Need uniform coverage with fewer samples than random | 20–50 configs |
| **Successive halving / Hyperband** | Each config has an early-progress signal (loss curve) | High-throughput early eval + long final |

Justify the choice in one sentence in the report header. If Bayesian-inspired, state which surrogate heuristic (EI-like pick of best-margin neighbors, UCB with tunable c, etc.) — no black-box "we used Bayesian" claims.

### 3. Generate config list

Emit a single YAML file at `experiments/dse/{name}/configs.yaml`:

```yaml
dse_name: "{name}"
strategy: "random"          # grid | random | bayes | lhs | hyperband
seed: 42                    # for reproducibility of the sampler itself
plan_ref: "experiments/{exp_name}/plan.md"
objective:
  metric: "val/accuracy"
  direction: "max"
constraints:
  max_total_gpu_h: 200
  max_per_config_gpu_h: 6
  early_stop:
    metric: "val/loss"
    patience_epochs: 3
configs:
  - name: "dse-001"
    params: {lr: 0.001, hidden: 256, dropout: 0.1, optimizer: "AdamW"}
  - name: "dse-002"
    params: {lr: 0.01, hidden: 512, dropout: 0.2, optimizer: "AdamW"}
  # ...
```

- `name` is sequential within this sweep — `{dse_name}-{NNN}`.
- Every `params` dict must be a superset of the plan's controlled variables; only the DSE axes differ.
- For Bayesian-inspired / sequential: emit only the first batch (typically 5–10 configs) and regenerate after results land.

### 4. Execute iteratively

For each batch:

1. Dispatch each config via `experiment-run` (passing `--exp-group {dse_name}` so downstream tooling can group results).
2. Poll via `experiment-monitor` or wait on completion.
3. Collect completed configs into `experiments/dse/{name}/results.csv` with schema: `name, {all params}, {primary metric}, {secondary metrics}, status, wall_clock, gpu_h`.
4. **Early-stop underperformers**: if a config's val metric at 20% of training is ≥ 2σ below current-best-at-same-fraction, kill it and mark `status: early_stopped`. This is the biggest budget-saver; don't skip it unless the metric is too noisy at checkpoint granularity.
5. For sequential strategies: inspect `results.csv`, pick next configs from the surrogate's most-promising region, return to step 3.
6. Stop when any triggers: budget exhausted / no improvement over last B batches (B = 3 default) / user halt.

### 5. Analyze + report

After the sweep concludes, produce `experiments/dse/{name}/report.md`:

```markdown
---
dse_name: "{name}"
plan_ref: "experiments/{exp_name}/plan.md"
strategy: "{strategy}"
configs_completed: N
configs_early_stopped: M
total_gpu_h: {T}
best_config: "{name of winning config}"
best_{metric}: {value}
baseline_{metric}: {value, from plan.md}
---

## Setup
{search space, strategy justification, constraints, any mid-sweep changes}

## Best Configuration
- **Config**: `{name}`
- **Params**: {full param dict}
- **Primary metric**: {value} ({Δ} vs baseline)
- **Secondary metrics**: {table}

## Sensitivity Analysis
For each DSE axis:
- **{param}**: {low / medium / high} sensitivity — {evidence: range of primary metric as param varies with others fixed at best}
- Ranked bar chart: `experiments/dse/{name}/figures/sensitivity.pdf` (via `paper-assets` figure mode)

## Search Landscape
- 2D projections onto the top-2 most sensitive axes (pairplot or heatmap)
- Trajectory (for sequential strategies): primary metric vs config index
- `experiments/dse/{name}/figures/landscape.pdf` (via `paper-assets` figure mode)

## Pareto Frontier (if applicable)
{primary metric vs cost / secondary, with non-dominated configs marked}

## Recommendations
1. **Promote** `{best_config}` to `experiment-run` for full-scale final runs with seeds.
2. **Drop** {list of parameters with negligible sensitivity} from future sweeps.
3. **Narrow** ranges on {list of parameters where optimum is at a boundary — extend the range next sweep}.

From Shiien/Self-Evolved-Research-Framework, MIT.

More from Shiien/Self-Evolved-Research-Framework