diagnose

Files (2)

• SKILL.md
• hitl-loop.template.sh · scripts/

SKILL.md (first lines)

# Diagnose

A discipline for hard bugs. Skip phases only when explicitly justified.

When exploring the codebase, use whichever domain glossary the project has (`AGENTS.md`, `~/.aigentry/CONSTITUTION.md` — installed by devkit bootstrap; skip if absent) to get a clear mental model of the relevant modules, and check ADRs in `docs/adr/` for the area you're touching.

## Where this skill fits

| Situation | Use |
|---|---|
| First-pass diagnosis on an active bug or regression | **`diagnose`** (this skill) — reproduce-first loop |
| Loop fails / dead end / multiple failed hypotheses | **`superpowers:systematic-debugging`** — deeper structured framework |
| Need cross-LLM second opinion after 3 failed attempts | Delegate to Codex/Gemini per your projects-root `CLAUDE.md` AI 작업 원칙 (no such file → just delegate) |
| Diagnosis target is "what should we build" not "why does it break" | Wrong skill — use `grill-with-adr` or `brainstorming` |

This skill is also the canonical playbook for the **analyst session** role (AGENTS.md role table: "runtime 분석 (판단)"). Analyst takes already-collected runtime evidence (logs, stack traces, datasets) and runs this loop. Analyst does **not** collect external info — that is dustcraw's job.

## Evidence-based commitment

Per memory `feedback_evidence_based_bugfix.md`: **no guess patches.** Every fix lands on a chain of logs, repro output, or upstream-issue references. If you cannot trace the fix to evidence, you have not finished Phase 3 — go back.

## Phase 1 — Build a feedback loop

**This is the skill.** Everything else is mechanical. If you have a fast, deterministic, agent-runnable pass/fail signal for the bug, you will find the cause — bisection, hypothesis-testing, and instrumentation all just consume that signal. If you don't have one, no amount of staring at code will save you.

Spend disproportionate effort here. **Be aggressive. Be creative. Refuse to give up.**

### Ways to construct one — try them in roughly this order

1. **Failing test** at whatever seam reaches the bug — unit, integration, e2e.
2. **Curl / HTTP script** against a running dev server.
3. **CLI invocation** with a fixture input, diffing stdout against a known-good snapshot.
4. **Headless browser script** (Playwright / Puppeteer) — drives the UI, asserts on DOM/console/network.
5. **Replay a captured trace.** Save a real network request / payload / event log to disk; replay it through the code path in isolation.
6. **Throwaway harness.** Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
7. **Property / fuzz loop.** If the bug is "sometimes wrong output", run 1000 random inputs and look for the failure mode.
8. **Bisection harness.** If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can `git bisect run` it.
9. **Differential loop.** Run the same input through old-version vs new-version (or two configs) and diff outputs.
10. **HITL bash script.** Last resort. If a human must click, drive _them_ with a structured loop script so the loop is still structured. Captured output feeds back to you. Start from `scripts/hitl-loop.template.sh` in this skill directory (`~/.claude/skills/diagnose/scripts/hitl-loop.template.sh` once installed) — copy it, edit the `step`/`capture` calls, run it.

Build the right feedback loop, and the bug is 90% fixed.

### Iterate on the loop itself

Treat the loop as a product. Once you have _a_ loop, ask:

- Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
- Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
- Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)

A 30-second flaky loop is barely better than no loop. A 2-second deterministic loop is a debugging superpower.

### Non-deterministic bugs

The goal is not a clean repro but a **higher reproduction rate**. Loop the trigger 100×, parallelise, add stress, narrow timing windows, inject sleeps. A 50%-flake bug is debuggable; 1% is not — keep raising the rate until it's debuggable.

### When you genuinely cannot build a loop

Stop and say so explicitly. List what you tried. Ask the user (or, if dispatched, the orchestrator via `telepty inject`) for: (a) access to whatever environment reproduces it, (b) a captured artifact (HAR file, log dump, core dump, screen recording with timestamps), or (c) permission to add temporary production instrumentation. Do **not** proceed to hypothesise without a loop.

Do not proceed to Phase 2 until you have a loop you believe in.

## Phase 2 — Reproduce

Run the loop. Watch the bug appear.

Confirm:

- [ ] The loop produces the failure mode the **user** described — not a different failure that happens to be nearby. Wrong bug = wrong fix.
- [ ] The failure is reproducible across multiple runs (or, for non-deterministic bugs, reproducible at a high enough rate to debug against).
- [ ] You have captured the exact symptom (error message, wrong output, slow timing) so later phases can verify the fix actually addresses it.

Do not proceed until you reproduce the bug.

## Phase 3 — Hypothesise

Generate **3–5 ranked hypotheses** before testing any of them. Single-hypothesis generation anchors on the first plausible idea.

Each hypothesis must be **falsifiable**: state the prediction it makes.

> Format: "If <X> is the cause, then <changing Y> will make the bug disappear / <changing Z> will make it worse."

If you cannot state the prediction, the hypothesis is a vibe — discard or sharpen it.

**Show the ranked list to the user (or orchestrator) before testing.** They often have domain knowledge that re-ranks instantly ("we just deployed a change to #3"), or know hypotheses they've already ruled out. Cheap checkpoint, big time saver. Don't block on it — proceed with your ranking if AFK.

When stuck on the same hypothesis class for 3 attempts, stop and delegate to a different LLM per AI 작업 원칙 → "3회 실패 시 Multi-LLM 위임". Same LLM tends to anchor on the same wrong frame; a fresh LLM with upstream-search bias often resolves it in one pass.

## Phase 4 — Instrument

Each probe must map to a specific prediction from Phase 3. **Change one variable at a time.**

Tool preference:

1. **Debugger / REPL inspection** if the env supports it. One breakpoint beats ten logs.
2. **Targeted logs** at the boundaries that distinguish hypotheses.
3. Never "log everything and grep".

**Tag every debug log** with a unique prefix, e.g. `[DEBUG-a4f2]`. Cleanup at the end becomes a single grep. Untagged logs survive; tagged logs die.

**Perf branch.** For performance regressions, logs are usually wrong. Instead: establish a baseline measurement (timing harness, `performance.now()`, profiler, query plan), then bisect. Measure first, fix second.

## Phase 5 — Fix + regression test

Write the regression test **before the fix** — but only if there is a **correct seam** for it.

A correct seam is one where the test exercises the **real bug pattern** as it occurs at the call site. If the only available seam is too shallow (single-caller test when the bug needs multiple callers, unit test that can't replicate the chain that triggered the bug), a regression test there gives false confidence.

**If no correct seam exists, that itself is the finding.** Note it. The codebase architecture is preventing the bug from being locked down. Flag this for the next phase.

If a correct seam exists:

1. Turn the minimised repro into a failing test at that seam.
2. Watch it fail.
3. Apply the fix.
4. Watch it pass.
5. Re-run the Phase 1 feedback loop against the original (un-minimised) scenario.

## Phase 6 — Cleanup + post-mortem

From dmsdc-ai/aigentry-devkit, MIT.

More from dmsdc-ai/aigentry-devkit