write-tests

Files (1)

• SKILL.md

SKILL.md (first lines)

# Write Tests

A good test fails **only** when real behavior breaks, and passes through every
refactor or config change that preserves it. Most bad tests fail the opposite
way: red on harmless changes, green while the real path is broken. Every rule
below serves that one goal.

Default to end-to-end or system-level tests for regression coverage; use a unit test only for specific, complex business logic when fast feedback will materially speed development.

For deciding whether coverage adds independent proof, where it belongs, or
which existing tests can go, use [audit-tests](../audit-tests/SKILL.md).

## Workflow: tracer bullets, not a batch

1. **Write ONE test at a time.** Assert first, watch it go red on the un-fixed
   code, make the code earn green, learn, then write the next. Never a batch up
   front: a batch written against *imagined* behavior pins what you guessed —
   those tests pass when the mechanism breaks and fail when it's fine. Each
   green cycle tells you what the next test should actually assert.
2. **Iterate on the fastest focused runner** (one file, one test name), and run
   the full suite only as a final gate before handing off. Check the exit code,
   not just the output — a runner that prints nothing and a green run look the
   same. On failures, read EVERY red test before fixing one; they often share a
   root cause.
3. **Before calling it done**, prove the test can fail (below) and walk the
   review checklist.

## What to assert

- **Observable behavior through the outermost practical entry point** — return
  values, exit codes, persisted rows, HTTP responses, rendered output — never
  which internal functions ran or how a value is computed. A test on the public
  surface survives a rewrite of everything underneath; a test that reaches into
  internals breaks on every refactor and pins implementation, not behavior.
  Use an isolated unit test only when the logic is specific and complex enough
  that its faster feedback will materially speed development.
- **Nothing the compiler already guarantees.** A test that re-asserts a type
  signature — field shapes, rejected argument types — can only fail if the
  compiler failed first. Spend the budget on business rules, arithmetic,
  branching, ordering, edge cases, side effects.
- **Actual values, not collection sizes.** For dedup/normalize/idempotency
  paths, `length == 1` passes even when normalization is broken; also assert
  the stored value equals the expected canonical form.
- **Keep system scenarios small.** Use the minimum data and mechanisms needed
  to prove the end-to-end behavior without entangling unrelated claims.
- **The design contract, not current behavior.** When a test goes red, the
  reflex is to re-measure and pin the new number — resist it: a bar calibrated
  to whatever the code currently does silently encodes bugs as baseline. Write
  the assert from the stated contract and make the code earn it; if no contract
  exists, that's a question for the owner, not a number to measure-and-pin.
  Recalibrating is legitimate only when the contract itself changed.
- **Deterministic claims tight, stochastic claims as distributions.** An
  invariant that holds every run gets an exact threshold. Anything noisy or
  tuning-dependent ("A usually beats B", "load balances evenly") must be
  asserted over a set of runs/seeds as a band — a single-sample pin on a
  stochastic outcome is not a weak test, it is a **blind** one: it certifies
  whatever the lucky sample did and can mask a systematic bias for months. If
  you must assert a noisy differential, widen the margin and name it
  chaos-marginal in a comment.

## Seams and mocks

- **Don't couple tests to config — mock the seam.** A test keyed to a live
  config value (which flag is on, which tier is default) breaks when someone
  legitimately edits config. The fix is never `skipIf`/conditionals — a skipped
  assertion hides the coupling and stops covering the path. Feed the function
  fixed inputs, or stub the lookup so fixture ids resolve to fixed values.
  Litmus: "would this break if a config value changed with no logic change?"
- **A test sets the bounds it depends on.** Tuning data (unit stats,
  ranges, prices, timings) is guaranteed to change: it is retuned all the
  time, and a test that reads it, or hard-codes today's value, breaks on every
  retune. So a boundary test (in range or out, seen or hidden, fits or
  overflows) uses a fake whose numbers the test itself defines, and probes
  either side of them. A test of anything else runs at the smallest setup that
  shows the behaviour, far from any bound: it survives retuning and runs
  faster.
- **Don't over-mock — every mock is a frozen assumption.** Mock at the system's
  edges (network, clock, filesystem, third-party SDK, config lookup), never
  internal collaborators. Stubbing an internal hard-codes its current contract;
  refactor it and the test lies. Never make "was called with" the primary
  assertion when an observable outcome exists. Mocking three internal modules
  to test one function means: test one layer up, where they're real.
- **Harnesses must wire the system the way production does.** If a bug only
  showed up against a real environment, the harness skipped an input production
  always sets — fix the harness, don't just fix the bug.

## Control variables and probes

- **One variable per comparison.** Pin everything else — same seed, same
  counts, same configuration on both arms; disable mechanisms that aren't the
  subject. When a comparison test breaks, first ask whether an unrelated
  mechanism leaked into the experiment before touching the code under test.
- **Validate, don't assume.** Never reason your way to a conclusion about
  *cause* — which path fired, where the failures come from — and act on it.
  Write a throwaway probe that reads public state and prints; the plausible
  story is wrong often enough to burn a session, and one probe redirects the
  whole effort. Probes are scratch: put them where they cannot be committed,
  delete them the moment the question is answered, or promote them into a real
  test if the behavior deserves a permanent guard.

## Tests run on someone's live machine

A suite that grabs the desktop is a suite people stop running. Whatever the test
needs, it takes the least intrusive form of it:

- **Never take focus.** No activating the app, no `makeKeyAndOrderFront`, no
  raising, no moving the pointer, no simulated clicks into the session, no audio
  playback. A UI a person must not be interrupted by is still fully testable:
  build the window, drive its model, and assert on the rendered view. Where a
  screenshot is the evidence, capture the window's own image offscreen rather
  than photographing the screen. Keep activation on the code path a person
  triggers, and exercise it by calling the handler, not by launching the app
  repeatedly.
- **Write nowhere the person keeps their own state.** Point every test at a
  scratch home, a scratch preferences domain or suite, and scratch install
  locations. A test must never modify the real config, library, login items or
  installed applications. Production may read an override the tests set; it must
  behave normally when that override is absent.
- **Batch anything unavoidably visible.** If a check genuinely needs a real

From dzhng/skills, MIT.

More from dzhng/skills