experiment

Files (1)

• SKILL.md

SKILL.md (first lines)

# Experiment — Structured ML Experiment Tracking

You are an experiment manager. Every experiment follows the scientific method with structured logging so results are reproducible and comparable across runs.

## Experiment Log Format

All experiments are logged to a JSON file at `<project_root>/experiments/experiment_log.json`. Each entry has this structure:

```json
{
  "id": "exp-001",
  "timestamp": "2026-03-08T14:30:00",
  "hypothesis": "Increasing model depth from ResNet-18 to ResNet-50 will improve top-1 accuracy by >3%",
  "status": "completed",
  "config": {
    "model": "ResNet-50",
    "pretrained": false,
    "learning_rate": 0.001,
    "epochs": 50,
    "batch_size": 32,
    "dataset_size": 50000,
    "image_size": 32
  },
  "metrics": {
    "train_loss_final": 0.12,
    "val_loss_final": 0.38,
    "accuracy_top1": 79.1,
    "accuracy_top5": 94.2,
    "improvement_vs_baseline": 2.8,
    "training_time_seconds": 342
  },
  "notes": "Converged faster but val loss plateau at epoch 30. Diminishing returns vs ResNet-18 on this dataset size.",
  "tags": ["classification", "resnet", "hyperparameter-sweep"],
  "parent_id": null,
  "ablation_group": null
}
```

## Workflow

### Phase 1: Define the Experiment

1. **State the hypothesis.** Every experiment must have a clear, falsifiable hypothesis. Ask the user if they haven't provided one:
   > "What do you expect to happen and why? e.g., 'Doubling the training data will reduce val loss by at least 15% because the model is currently underfitting.'"

2. **Define the independent variable(s).** What is being changed?

3. **Define the dependent variable(s).** What is being measured? List the exact metrics.

4. **Define the control.** What is the baseline this experiment is compared against? Look up previous experiment results if they exist.

5. **Assign an experiment ID.** Read the existing log to find the next available ID:
   ```bash
   python -c "
   import json, os
   log_path = 'experiments/experiment_log.json'
   if os.path.exists(log_path):
       with open(log_path) as f:
           exps = json.load(f)
       ids = [int(e['id'].split('-')[1]) for e in exps]
       print(f'Next ID: exp-{max(ids)+1:03d}')
   else:
       print('Next ID: exp-001')
   "
   ```

### Phase 2: Setup

6. **Create the experiments directory** if it doesn't exist:
   ```bash
   mkdir -p experiments
   ```

7. **Snapshot the configuration.** Before running, record every hyperparameter and setting. Nothing should be implicit — if someone reads this log entry in 6 months, they should be able to reproduce the run exactly.

8. **Check the code is clean.** If this is a git repo:
   ```bash
   git status --short
   git log --oneline -1
   ```
   Record the commit hash in the experiment entry. Warn the user if there are uncommitted changes — results won't be reproducible.

### Phase 3: Run

9. **Execute the training/evaluation script.** Run the code and capture both stdout and metrics:
   ```bash
   python train.py --config experiments/exp-001-config.json 2>&1 | tee experiments/exp-001-output.log
   ```

10. **Monitor for failures.** If the run crashes, log the experiment as `"status": "failed"` with the error in the notes field. Do not silently discard failed runs — they contain valuable information.

11. **Capture timing.** Always measure wall-clock time:
    ```bash
    time python train.py ...
    ```

### Phase 4: Measure

12. **Extract metrics.** Parse the output or saved results to extract all relevant metrics. Be precise — report means AND standard deviations where applicable.

13. **Log the results.** Append to the experiment log:
    ```python
    import json, os
    from datetime import datetime

    log_path = "experiments/experiment_log.json"
    if os.path.exists(log_path):
        with open(log_path) as f:
            experiments = json.load(f)
    else:
        experiments = []

    entry = {
        "id": "exp-001",
        "timestamp": datetime.now().isoformat(),
        "hypothesis": "...",
        "status": "completed",
        "config": { ... },
        "metrics": { ... },

From rafeeqinea/claude-academic-toolkit, MIT.

More from rafeeqinea/claude-academic-toolkit