ml-coding

Files (6)

• SKILL.md
• delegation.md · references/
• diagnosis-discipline.md · references/
• distributed-training.md · references/
• documentation-long-running.md · references/
• reviewing-research-code.md · references/

SKILL.md (first lines)

# ML Coding

Write and diagnose machine learning research the way a lab does: let errors surface, make every experiment trustworthy, and prefer the framework's existing extension point over a custom shim.

## Research code is not product code

- A pipeline is verified once, offline: after that, strip debug/test/validation code out of the production path. For example, per-item numeric checks, write-then-read-back comparison, recomputed hashes, and re-derived invariants all re-establish a conclusion you already hold, and they cost real throughput.
- Validate the claim, not every possible environment: test the behavior the change is intended to preserve or improve. For example, cover additional GPUs, machines, or checkpoints only when the claim depends on them.
- Smoke testing means it runs: Finding a bug during a real run and fixing it there is the expected path; no mechanism arrives complete on the first attempt. Do not expand a smoke check into a validation campaign. When a strict criterion turns out to be unreachable in principle — because the reference path itself is nondeterministic — say so and replace it with a decidable one instead of chasing it.
- Research code is still for humans to read (collaborators, future you). Name things for what they mean and keep interfaces ordered; let comments carry the design intent that the deliberate absence of assertions would otherwise leave undocumented. Comments explain why a non-obvious choice exists, not what the next line literally does.

### Fail fast, fail loud

Let missing fields, bad shapes, and wrong configs crash at the source; keep a check only when its absence would fail silently.

- Accept natural errors: missing attributes raise AttributeError, shape mismatches raise from PyTorch, bad configs crash. Do not wrap these with custom messages.
- No defensive programming for ourselves: code and configs are written by us. Correct values live in yaml/docs, not in runtime checks and assertions. Avoid similar conditional checks,assertions, and exception raising. For example, do not validate caller inputs like `training_mode` legality, hyperparameter ranges.
- No `hasattr`/`getattr` probe chains for structures you know: Access the real path; a missing field should raise KeyError naturally. Avoid `.get(key, default)` for fields that must exist, and never stack fallbacks — they turn a missing field into a silently wrong value.
- No default parameter values inside functions. Defaults live only in config, passed explicitly from outside. Internal defaults hide the effective value, and a missed config update silently uses the wrong one.
- The only and sole exception checks worth keeping are those whose absence fails silently: it does not crash but produces wrong data or wrong conclusions. When keeping one, comment what silent failure it prevents.

## Reuse before building

- Match existing code: style, call patterns, structure. The official precedent in the same repo is the best spec — copy how it is done, do not reinvent it.
- When subclassing, verify the parent's full capability stays online, not just that your new code works. One missed parent method is a missing feature in single process; in distributed training it can break cross-rank consistency.
- Prefer the framework's official extension point (callbacks, hooks) over monkey-patching internal methods. Custom trace launchers are a recurring source of the hard-to-diagnose failures the framework already has a slot for.
- Import optional analysis dependencies inside the function that uses them, so a missing plotting or stats package cannot break training or evaluation. A test that constructs DeepSpeed/FSDP arguments must restore process-level state afterwards — those constructors mutate global accelerator state, and leaking it makes a later unrelated test fail in a way that points nowhere near the cause.

## Repository hygiene: what may be committed

- Committed configs are general and reusable. A config whose name encodes a step count, an experiment arm, a task instance, a machine, or a date is a run record, not a config — it does not belong in the tracked config directory.
- A one-off run keeps its own directory outside version control and overrides parameters there. Overriding in a run directory is the supported way to pin the specifics of one experiment. One-off debug/verification code (prints, counters, comparisons, `--contract-only` switches) is deleted once the conclusion is written up.
- When a hyperparameter change becomes the long-term default, edit the general config so the new value is what everyone reproduces. Do not express a lasting decision by adding another variant file next to the old one. The same rule covers entry points: no one-off launcher scripts, no `.sh` wrappers named after a single run, no backup copies (`*.bak*`) — Git already holds the history, and a stale backup can still be executed and produce wrong results. Two costs drive this, and both grow silently: a directory of near-identical variants stops being readable by humans, and a fresh agent inheriting the repository cannot tell which file is authoritative.
- Tests assert the general parameter contract. Generate any unusual combination as a temporary file inside the test; never require the repository to keep an experiment's config alive so a test can pass.
- A run directory that reuses an existing output path is worse than untidy: a resumable pipeline infers progress from artifacts already on disk, so it can skip stages, consume old evidence, and still report success. New run identity, new output path.

## References

Load the matching reference when the situation applies.

| Situation | Reference |
|---|---|
| Distributed deadlocks, hangs, or cross-rank inconsistency | [distributed-training.md](references/distributed-training.md) |
| Diagnosing silent-wrong results or a complex failure | [diagnosis-discipline.md](references/diagnosis-discipline.md) |
| Reviewing a diff, a PR, or a delegated agent's work | [reviewing-research-code.md](references/reviewing-research-code.md) |
| Maintaining docs or handing off a long-running project | [documentation-long-running.md](references/documentation-long-running.md) |
| Delegating a rewrite or verification task to another agent | [delegation.md](references/delegation.md) |

## Some Coding Principles (when applicable)

- IPO (Input, Process, Output) principle: Pin the input and the output; the process between them is only a means. Optimizing a module or framework path means the final artifact and its declared parameters stay identical — intermediate formats, staging layout, decode path, chunking, scheduling order are not part of the contract and need not match the previous implementation. Compare the end artifact once; do not prove intermediate parity, and do not recompute already-usable results for format uniformity.

From Rvosuke/Skills, MIT.

More from Rvosuke/Skills