how-to-experiment

Files (1)

• SKILL.md

SKILL.md (first lines)

# How To Experiment

Use this skill whenever work involves measuring whether an idea is good: model training,
classifier accuracy, embedding quality, search relevance, threshold tuning, UI heuristics,
performance changes, ablations, or comparisons between approaches.

The purpose is not bureaucracy. It is to prevent fast prototype loops from producing numbers
that look convincing but do not generalize. Deep Cuts has already hit this failure mode: the
SAX sequence tagger reported **99.27% accuracy on the same 740 tracks used for training** in the
[session-2026-06-06-sax-transformer](http://gitea:3000/botfam/deep-cuts/wiki/session-2026-06-06-sax-transformer)
wiki page; the follow-up in
[session-2026-06-06-sax-transformer-future-ideas](http://gitea:3000/botfam/deep-cuts/wiki/session-2026-06-06-sax-transformer-future-ideas)
(code and sample data now live in `tools/sax-transformer/`)
correctly marked that number as untrustworthy until evaluated on a held-out split.

## Minimum Protocol

Before running the experiment, write down:

1. **Question**: the specific claim being tested.
2. **Unit of split**: usually track, album, artist, or library. Never split by segment/window if
   segments from the same track can leak into both train and test.
3. **Metric**: the primary metric and any acceptance threshold.
4. **Baseline**: the current production behavior, simple heuristic, or prior model.
5. **Frozen evaluation set**: what data is reserved for final reporting.
6. **Artifacts**: scripts, command lines, seeds, dataset versions, and output files.

If any of these are missing, call the run exploratory and do not make generalization claims.

## Data Splits

Use three sets when tuning is involved:

- **Train**: fit model weights, thresholds, centroids, clusters, or prompt examples.
- **Validation**: choose hyperparameters, early stopping, thresholds, prompt variants, UI
  constants, and feature combinations.
- **Test / holdout**: final one-time estimate. Do not tune on it.

Default split for enough data: **70% train / 10% validation / 20% test**, stratified when useful
by genre, artist, label class, duration bucket, or library source.

For small data:

- Prefer grouped cross-validation over repeatedly peeking at one tiny holdout.
- Keep a final untouched holdout if the result will drive production behavior.
- Report uncertainty: per-class counts, confidence intervals, or "N is too small" notes.

For music data, prevent leakage aggressively:

- Split by **track** at minimum.
- Split by **artist or album** when evaluating generalization to unseen music styles.
- Keep derived windows, SAX segments, embeddings, lyrics labels, and cached pass outputs on the
  same side of the split as their source track.
- If near-duplicates or alternate masters exist, keep them in the same split.

## Test Set Rules

The test set is not a dashboard knob.

- Do not inspect test errors while choosing parameters.
- Do not rerun many variants and report the best test score.
- Do not add "just one fix" after seeing test failures without moving back to validation and
  creating a new final holdout.
- If the test set was used for decisions, say so and rename it to validation.

Acceptable use of test results:

- Final report after the approach is frozen.
- Regression check for a previously frozen production model.
- External benchmark evaluation with no fine-tuning on that benchmark.

## Baselines And Ablations

Every experiment needs a comparison.

- **Baseline**: current app behavior, a simple rule, random/majority class, or previous model.
- **Ablation**: remove one suspected shortcut or feature at a time.
- **Sanity check**: shuffle labels, use random embeddings, or test a trivial predictor when
  appropriate.

Deep Cuts example: for structural labels, `position` can be a shortcut. A model using
`energy + rep_score + position` must be compared against `energy + rep_score` without
`position` before claiming it learned musical structure.

## Metrics

Pick the metric that matches the task, not the one that is easiest to compute.

- Classification: report overall accuracy plus per-class precision/recall/F1 and class counts.
- Imbalanced labels: prefer macro F1, balanced accuracy, or per-class recall over raw accuracy.
- Ranking/search: use fixed query sets with precision@K, recall@K, MRR, nDCG, or blinded human
  judgments.
- Segmentation/boundaries: use boundary F-measure with a stated tolerance when comparing to
  music-structure benchmarks; fixed-frame accuracy may be useful internally but is not the
  standard public claim.
- Performance: separate cold start, warm run, per-track latency, throughput, memory, and model
  load time.

When a metric is a proxy, say what it does not prove.

## Reporting Template

Use this shape in docs, session logs, or handoffs:

```markdown
## Experiment: <name>

Question:
Dataset:
Split unit:
Split:
Leakage controls:
Baseline:
Variants:
Primary metric:
Secondary metrics:
Commands/scripts:
Result:
Decision:

From robertolupi/deep-cuts, AGPL-3.0.

More from robertolupi/deep-cuts