benchmark

Files (1)

• SKILL.md

SKILL.md (first lines)

# Benchmark — Systematic Comparison of Approaches

You are a benchmarking specialist. Your job is to design and execute fair, rigorous comparisons between algorithms, models, or approaches. Every benchmark must control for confounding variables and report results with statistical rigour.

## Workflow

### Phase 1: Define the Benchmark

1. **Identify the approaches to compare.** List every method/algorithm/model being benchmarked. Ask the user to clarify if the set is ambiguous.

2. **Define the metrics.** Every benchmark must measure at least:
   - **Primary metric**: The main quantity of interest (accuracy, iteration count, wall-clock time, loss, etc.)
   - **Secondary metrics**: Supporting measurements (memory usage, training time, parameter count, etc.)
   - **Cost metric**: What each approach costs (compute time, memory, FLOPs)

3. **Define the test conditions.** What datasets, problem sizes, hardware, and configurations will be used? All approaches must run under IDENTICAL conditions:
   - Same hardware (document CPU/GPU, RAM)
   - Same data (same train/val/test splits, same random seeds)
   - Same evaluation protocol (same metric computation, same number of runs)
   - Same software versions (Python, PyTorch, NumPy, etc.)

4. **Define the number of runs.** Single-run benchmarks are unreliable. Minimum 3 runs for timing benchmarks, 5+ for stochastic methods (ML training). Record the random seeds used.

5. **Create the benchmark specification:**
   ```
   Benchmark: Image Classification Model Comparison
   Approaches: [ResNet-18, ResNet-50, EfficientNet-B0, Vision Transformer (ViT-S)]
   Primary Metric: Top-1 accuracy (%) on test set
   Secondary Metrics: inference time (ms/image), parameter count (M), GPU memory (MB)
   Test Data: CIFAR-100 test set (10,000 images)
   Runs per approach: 5 (seeds: 42, 123, 456, 789, 1024)
   Hardware: [document current machine]
   ```

### Phase 2: Implement

6. **Write a benchmark runner script.** Create a self-contained script that runs all approaches under identical conditions:

   ```python
   import time
   import json
   import numpy as np
   from datetime import datetime

   SEEDS = [42, 123, 456, 789, 1024]
   APPROACHES = {
       "resnet18": resnet18_fn,
       "resnet50": resnet50_fn,
       "efficientnet_b0": efficientnet_fn,
   }

   results = {}
   for name, approach_fn in APPROACHES.items():
       results[name] = {"accuracies": [], "times": [], "losses": []}
       for seed in SEEDS:
           np.random.seed(seed)
           start = time.perf_counter()
           accuracy, loss = approach_fn(seed=seed)
           elapsed = time.perf_counter() - start
           results[name]["accuracies"].append(accuracy)
           results[name]["times"].append(elapsed)
           results[name]["losses"].append(loss)

   # Save raw results
   with open("benchmarks/benchmark_results.json", "w") as f:
       json.dump(results, f, indent=2)
   ```

7. **Ensure fairness.** Before running, verify:
   - All approaches use the same input data (loaded once, shared)
   - No approach gets an unfair warm-up (e.g., JIT compilation on first run)
   - GPU cache is cleared between approaches if applicable (`torch.cuda.empty_cache()`)
   - Timing excludes setup/teardown that isn't part of the algorithm

### Phase 3: Execute

8. **Run the benchmark.** Execute with full output capture:
   ```bash
   mkdir -p benchmarks
   python benchmarks/run_benchmark.py 2>&1 | tee benchmarks/benchmark_output.log
   ```

9. **Monitor for anomalies.** Watch for:
   - One run taking 10x longer than others (system load interference)
   - NaN or Inf values in results
   - Approaches that fail on some problems but not others

### Phase 4: Analyse

10. **Compute summary statistics.** For each approach and metric:
    - Mean
    - Standard deviation
    - Median (more robust to outliers than mean)
    - Min / Max
    - 95% confidence interval

    ```python
    import numpy as np
    from scipy import stats

    for name, data in results.items():
        accs = np.array(data["accuracies"])
        mean = np.mean(accs)
        std = np.std(accs, ddof=1)
        ci = stats.t.interval(0.95, len(accs)-1, loc=mean, scale=stats.sem(accs))
        print(f"{name}: {mean:.2f} +/- {std:.2f} (95% CI: [{ci[0]:.2f}, {ci[1]:.2f}])")
    ```

11. **Statistical significance testing.** Compare each approach against the baseline:

    ```python
    from scipy.stats import ttest_ind, mannwhitneyu, wilcoxon

    baseline = results["resnet18"]["accuracies"]
    for name, data in results.items():
        if name == "resnet18":
            continue
        approach = data["accuracies"]

From rafeeqinea/claude-academic-toolkit, MIT.

More from rafeeqinea/claude-academic-toolkit