diagnosing-bugs

Files (5)

• SKILL.md
• evals.json · evals/
• REFERENCE.md · references/
• feedback-loops.md · references/
• hitl-loop.template.sh · scripts/

SKILL.md (first lines)

# Diagnosing Bugs

A discipline for hard bugs. Skip phases only when explicitly justified.

When exploring the codebase, read the project's domain glossary — `CONTEXT.md` unless the domain doc config points elsewhere — for a mental model of the modules involved, and check the ADRs covering the area you're touching, so a hypothesis doesn't contradict a decision already made.

## Redact

This skill has you show commands, outputs and captured artifacts. **Redact every secret first** — write `<REDACTED>` in its place. Build loops against env vars, so the credential stays in the environment rather than in what you show. Captured artifacts carry auth headers: quote only the lines that carry the signal.

**The whole skill in one breath:** build a **tight feedback loop** — one command that is *fast (seconds), deterministic, and red-capable* (goes red on this exact bug) — then minimize the repro and test hypotheses one variable at a time. Record a representative **permanent regression observed red** for this bug's reason before **production repair**; verify it green and rerun the original scenario afterward. If 5+ failures exist at once, run Multi-Failure Triage first; otherwise start at Phase 1.

## Multi-Failure Triage (when 5+ failures exist)

When multiple failures exist — broken test suites, migration fallout, dependency upgrades, non-TDD development sessions — **do not fix them one-by-one with full-suite runs between each fix.** That wastes feedback-loop time on tests you already know pass.

**Full suite is a gate, not a feedback loop.**

This procedure is tier-agnostic: it works for unit, integration, E2E, or any mix. Use the reference commands as templates — adapt to your project's tooling.

### Output capture rule

**`| tee [filename]` is the ONLY allowed method for capturing test output** — `tail`, `head`, and bare `>`/`>>` are forbidden (test-run rule: `testing-workflow` skill; the general rule for every command whose output you read is `construction-craft` Critical rule 8). Triage-specific consequence: **after running with `| tee`, read or `grep` the log file — never re-run the same pytest command just to inspect output.** The file already has everything; re-running wastes a full-suite cycle.

### Timeout guidelines

Set **both** timeout layers explicitly on every triage run — outer (harness budget) and inner (`--timeout` per test), outer > inner. The tier-by-tier floors live in the `testing-workflow` skill's timeout doctrine; for triage the floors that matter most are `--timeout 600` for E2E and an outer budget of 1 hour — a run killed by a default 60s timeout reports on the runner, not the tests, and corrupts your failure map.

---

### Step 0 — CALIBRATE (run once, before triage)

**Goal:** Ensure your failure map is trustworthy. Don't triage on corrupted data.

**Skip 0a and 0b if all of the following are true:**
- You just ran the full suite and the failure set matches what you expect.
- No recent changes to fixtures, models, or shared code.
- The failures span multiple unrelated modules (unlikely to be xdist contamination).

Skipping is safe because xdist contamination and flakes both produce **inconsistent** failure sets — if your set is stable across runs, the calibration step adds no value.

**0a. Verify parallel stability (if you suspect xdist issues).**

```
# Serial baseline
pytest tests/<tier>/ -n 1 --timeout 600 --tb=line -q | tee /tmp/serial_failures.log

# Parallel run
pytest tests/<tier>/ -n auto --dist loadgroup --timeout 600 --tb=line -q | tee /tmp/parallel_failures.log
```

Compare failure sets. If they differ → **xdist contamination detected.** Tests that fail under parallelism but pass serially are likely sharing state. Add `xdist_group` markers to force them onto the same worker, then re-calibrate. **Do not proceed to triage until serial and parallel failure sets match.**

**0b. Quarantine flakes.**

```
pytest tests/<tier>/ --timeout 600 --count=3 -q
```

Tests that fail inconsistently are **flakes, not bugs**. Quarantine them (`xfail`, skip, or fix separately). They inflate the failure count and corrupt your cascade map. A 50%-flake test masquerades as a real failure half the time.

**0c. Split by tier (if multiple tiers are failing).**

If failures span unit, integration, and E2E — **fix the fastest tier first.**

- **Unit tests** → run on host, feedback in seconds. Fix these first — fastest ROI.
- **Integration tests** → run in Docker, feedback in tens of seconds. Keep the container warm.
- **E2E tests** → run in Docker with browser, slowest. Fix last.

Fixing unit tests often collapses the E2E failure count dramatically. Don't touch E2E until faster tiers are green.

---

### Step 1 — MAP (one full run, serialized)

**Goal:** Build a reliable failure map. One expensive run, done right.

Run the full failing tier with serialized output:

```
# With xdist (recommended)
pytest tests/<tier>/ -n auto --dist loadgroup --timeout 600 --json-report -q | tee /tmp/failures_round_0.log

# Without xdist
pytest tests/<tier>/ --timeout 600 --json-report -q | tee /tmp/failures_round_0.log
```

**Save the output as `failures_round_0.json`.** You'll diff against this file in later rounds.

Read every traceback. Group failures by **root cause**, not by file:

| Grouping signal | Likely root cause |
|----------------|-------------------|
| Same exception type across files | Shared import, fixture, or model change |
| Same module in traceback | Broken signal handler, service, or utility |
| `AttributeError` / `ImportError` cluster | Renamed/moved function or missing migration |
| `AssertionError` with same expected value | Changed behavior or hardcoded expectation |
| Client sees wrong UI but server logs show success | Client-side state not updated, or HTMX/server-swap overwrote injected content |
| Async message ordering (client sends X, server processes Y first) | Race condition in queue/sync logic |
| Test passes serially but fails parallel, or vice versa | Timing dependency masked by execution order |

**Expect 38 failures → ~8-12 actual problems.** One fix cascades through its cluster.

### Quick relevance check (before deep triage)

Before triaging all failures, determine which are **caused by your changes**
vs **pre-existing**. This prevents wasting effort on unrelated bugs.

**Method: `git stash` + targeted re-run.**

```
# Run only the failures most likely related to your changes
# (same module, same consumer, same template family)
git stash
pytest tests/<related_tier>/ -k "test_a or test_b" -n 1 --timeout 300 -q | tee /tmp/before.log
git stash pop
pytest tests/<related_tier>/ -k "test_a or test_b" -n 1 --timeout 300 -q | tee /tmp/after.log
diff /tmp/before.log /tmp/after.log
```

From dekoza/oh-my-slop, Unlicense.

More from dekoza/oh-my-slop