diagnosing-bugs

Files (2)

• SKILL.md
• hitl-loop.template.sh

SKILL.md (first lines)

# Diagnosing Bugs

A discipline for hard bugs. Skip a phase only when you can say why.

Read the project's `GLOSSARY.md` (if any) for a clear model of the modules
involved, and check ADRs in the area.

**Redact as you go.** This loop shows commands, output, and captured artifacts.
Write `<REDACTED>` in place of every secret, keep credentials in environment
variables the loop reads, and quote only the lines of a captured artifact that
carry the signal. If redacted output isn't enough to diagnose, say so and ask.

## Phase 1: Build a feedback loop

This is the skill. With a **tight** pass/fail signal that goes **red** on this
bug, bisection, hypotheses, and instrumentation all just consume it. Without
one, no amount of reading code will find the cause. Spend disproportionate
effort here, and keep at it.

Ways to build one, roughly in this order:

1. A failing test at whatever seam reaches the bug: unit, integration, or
   end-to-end.
2. A curl or HTTP script against a running dev server.
3. A CLI run with a fixture input, diffing output against a known-good
   snapshot.
4. A headless browser script (Playwright, Puppeteer) asserting on the DOM,
   console, or network.
5. Replaying a captured trace: save a real request, payload, or event log and
   run it through the code path in isolation.
6. A throwaway harness: the smallest subset of the system (one service, stub
   dependencies) that reaches the bug's path in one call.
7. A property or fuzz loop for "sometimes wrong": many random inputs, looking
   for the failure mode.
8. A bisection harness when the bug appeared between two known states
   (commits, datasets, versions), so `git bisect run` can drive it.
9. A differential loop: the same input through the old and new version (or two
   configs), diffing the output.
10. A human-in-the-loop script, as a last resort when a person must click.
    Copy `${CLAUDE_SKILL_DIR}/hitl-loop.template.sh` into the repo's scratch space and
    edit its steps. The person runs it in their own terminal, since your shell
    has no terminal for them to type into: in Claude Code they type
    `! bash <path>` at the prompt, elsewhere they run it and paste the output
    back. Done when the `--- Captured ---` block is back in the conversation.

**Tighten it.** Make it faster (cache setup, narrow the scope), sharper (assert
the exact symptom, not "didn't crash"), and more deterministic (pin time, seed
randomness, isolate the filesystem and network). For a flaky bug, aim for a
higher reproduction rate rather than a clean repro: loop the trigger,
parallelise, add stress, narrow timing windows. A 50% flake is debuggable; 1% is not yet.

**If you cannot build one,** stop and say so, list what you tried, and ask for
access to an environment that reproduces it, a redacted captured artifact (HAR
file, log dump, core dump, timestamped recording), or permission to add
temporary production instrumentation.

Done when you can name **one command**, already run at least once (show it and
its redacted output), that is:

- **Red-capable:** it drives the real bug path and asserts the user's exact
  symptom, so it goes red on this bug and green once it is fixed.
- **Deterministic:** the same verdict every run, or for a flaky bug a pinned,
  high reproduction rate.
- **Fast:** seconds, not minutes.
- **Agent-runnable:** you can run it unattended, or, for the human-in-the-loop
  case, the person can run the script and hand back its output.

Reading code to build a theory before this command exists is the exact failure
this skill prevents. No red-capable command, no Phase 2.

## Phase 2: Reproduce and minimise

Run the loop and watch it go red. Confirm the failure is the one the user
described, not a nearby different one; that it reproduces across runs (or at a
debuggable rate); and capture the exact symptom so later phases can check the
fix against it.

Then cut inputs, callers, config, data, and steps one at a time, re-running
after each cut, keeping only what the failure needs. A minimal repro shrinks
the hypothesis space and becomes the regression test. Done when every
remaining element is load-bearing: removing any one turns the loop green.

## Phase 3: Hypothesise

Write 3-5 ranked hypotheses before testing any; a single hypothesis anchors on
the first plausible idea. Each must be falsifiable: "If X is the cause, then
changing Y makes the bug disappear and changing Z makes it worse." A
hypothesis with no prediction is a hunch: sharpen it or drop it.

Show the ranked list to the user before testing. They often know something
that re-ranks it at once ("we changed #3 yesterday"), or what's already been
ruled out. Proceed on your own ranking if they're away. Done when the ranked,
falsifiable list has been shown.

## Phase 4: Instrument

Each probe tests one prediction from Phase 3; change one variable at a time.
Prefer a debugger or REPL over logs, and targeted logs at the boundaries that
separate hypotheses over logging everything. Tag every debug line with one
unique prefix (`[DEBUG-a4f2]`) so cleanup is a single grep.

For performance regressions, measure before fixing: establish a baseline
(timing harness, profiler, query plan), then bisect.

Done when one hypothesis is confirmed by a probe and each of the others is
ruled out by one, or the evidence sends you back to Phase 3 with new
hypotheses.

Watch for a spiral: after three rounds of "still broken", stop changing code.
Name the assumption most likely to be wrong (the bug is where you think, the
loop tests what you think, the build is the one you changed), and ask the user
one diagnostic question before the next change.

## Phase 5: Fix at the root, with a regression test

A report names a symptom. Before editing, list every caller of the function
you're about to change and note which of them hit the same defect. Then fix
the defect where all those callers route through, at its cause, not where the
symptom shows: patching only the path the report names leaves the sibling

From dougborg/harness-kit, Apache-2.0.

More from dougborg/harness-kit