debugging-test-failures

Files (38)

• SKILL.md
• evals.json · evals/
• AGENTS.md · evals/fixtures/phasewave/scenario-1-code-bug/
• package.json · evals/fixtures/phasewave/scenario-1-code-bug/
• lint.js · evals/fixtures/phasewave/scenario-1-code-bug/scripts/
• test.js · evals/fixtures/phasewave/scenario-1-code-bug/scripts/
• type-check.js · evals/fixtures/phasewave/scenario-1-code-bug/scripts/
• session-display.js · evals/fixtures/phasewave/scenario-1-code-bug/src/lib/
• session-list.js · evals/fixtures/phasewave/scenario-1-code-bug/src/lib/
• feat-session-sort.md · evals/fixtures/phasewave/scenario-2-test-defect/.opencode/plans/
• AGENTS.md · evals/fixtures/phasewave/scenario-2-test-defect/
• package.json · evals/fixtures/phasewave/scenario-2-test-defect/
• lint.js · evals/fixtures/phasewave/scenario-2-test-defect/scripts/
• test.js · evals/fixtures/phasewave/scenario-2-test-defect/scripts/
• type-check.js · evals/fixtures/phasewave/scenario-2-test-defect/scripts/
• session-display.js · evals/fixtures/phasewave/scenario-2-test-defect/src/lib/
• session-list.js · evals/fixtures/phasewave/scenario-2-test-defect/src/lib/
• AGENTS.md · evals/fixtures/phasewave/scenario-3-weakening-pressure/
• package.json · evals/fixtures/phasewave/scenario-3-weakening-pressure/
• lint.js · evals/fixtures/phasewave/scenario-3-weakening-pressure/scripts/
• test.js · evals/fixtures/phasewave/scenario-3-weakening-pressure/scripts/
• type-check.js · evals/fixtures/phasewave/scenario-3-weakening-pressure/scripts/
• session-list.js · evals/fixtures/phasewave/scenario-3-weakening-pressure/src/lib/
• AGENTS.md · evals/fixtures/phasewave/scenario-4-flake/
• package.json · evals/fixtures/phasewave/scenario-4-flake/
• lint.js · evals/fixtures/phasewave/scenario-4-flake/scripts/
• test.js · evals/fixtures/phasewave/scenario-4-flake/scripts/
• type-check.js · evals/fixtures/phasewave/scenario-4-flake/scripts/
• session-list.js · evals/fixtures/phasewave/scenario-4-flake/src/lib/
• feat-session-sort.md · evals/fixtures/phasewave/scenario-5-design-defect/.opencode/plans/
• AGENTS.md · evals/fixtures/phasewave/scenario-5-design-defect/
• package.json · evals/fixtures/phasewave/scenario-5-design-defect/
• lint.js · evals/fixtures/phasewave/scenario-5-design-defect/scripts/
• test.js · evals/fixtures/phasewave/scenario-5-design-defect/scripts/
• type-check.js · evals/fixtures/phasewave/scenario-5-design-defect/scripts/
• session-display.js · evals/fixtures/phasewave/scenario-5-design-defect/src/lib/
• session-list.js · evals/fixtures/phasewave/scenario-5-design-defect/src/lib/
• vue-supabase.md · references/stacks/

SKILL.md (first lines)

# Debugging Test Failures

The diagnosis discipline of the core loop. Entered from an
`implementing-features` pass that hit its honest-stop (Step 6
non-convergence: N focused fix attempts did not converge), or
standalone ("CI is red", "this test started failing", "the e2e suite
is flaky"). The implement skill converges; this skill explains **why
something will not converge** and removes the cause.

A debug pass is **done** when the root cause is identified and stated in
one sentence, the fix is applied at the root cause, the previously-
failing command exits 0, **and** the full verification suite still
passes (no regression traded in). Or an honest escalation (below). "It
passes now but I'm not sure why" is not closure — that is a refuted
diagnosis.

## The debug pass

Copy this checklist and check off items as you complete them.

```
Debug Progress:
- [ ] 1. Reproduce (run the failing command; read the actual output)
- [ ] 2. Detect stack & load the matching stack reference
- [ ] 3. Form a root-cause hypothesis
- [ ] 4. Design + run the cheapest discriminating experiment
- [ ] 5. Classify the defect (CODE / TEST / DESIGN-PLAN / ENVIRONMENT)
- [ ] 6. Apply the class-specific fix
- [ ] 7. Regression guard (run the FULL suite)
- [ ] 8. Closure (diagnosis record) or escalation
```

### Step 1 — Reproduce before diagnosing

Run the failing command and read the **actual** output before forming
any hypothesis or touching any source. The reported symptom is a
summary; the failure output is the evidence. Two branches:

- **Reproduced** (the command fails on a clean run) → proceed to
  Step 2. Never edit from the reported symptom alone — the symptom
  points at where the assertion tripped, which is often downstream of
  the defect.
- **Cannot reproduce on a single run** → that is the **flake branch**,
  not a license to say "works for me." Rerun the command several times
  (or run the single test in isolation, then in the suite, then in
  reverse order — order/shared-state flakes surface here). A flake is a
  real defect of class 4 (environment/nondeterminism); diagnose it,
  do not dismiss it.

**Console-visible defects: scripted-browser repro first.** When the
failure output names a console error / warning / pageerror (or the
symptom is "errors in the console that should not be present"), run a
**standalone Playwright script** before hypothesizing: library API
(`chromium.launch`, not the test runner), `storageState` from the
project auth fixture, `page.on('pageerror')` + `page.on('console')`
listeners, walk the exact user flow, print the captures. This produced
ground truth in one run (stack + state dump) where component-harness
reproduction was misdirecting — dev-mode framework warnings can be
runtime defects that production builds strip, and teardown/close-path
crashes fire only when the journey actually closes what it opens
(validating-ui's journey-completeness rule is the same principle).
Capture the browser console during the failing step and look for
`Failed to fetch dynamically imported module` / a 500 on a lazily-
imported module — a real defect in the changed code, not infra.

### Step 2 — Detect stack & load the matching stack reference

Identify the project stack from the rules file (`AGENTS.md` /
`.opencode/agents.md`). If this skill bundles a matching stack
reference under `references/stacks/` (resolved against this skill's own
directory — not the project's), read it now and apply its debugging-time
concerns. If none exists, proceed generically and flag the gap to the
user in the handoff — a missing stack reference is a finding, not a
silent "I'll improvise."

Available stack references:

- vue-supabase → [references/stacks/vue-supabase.md](references/stacks/vue-supabase.md)
  Read when the project is Vue 3 + Pinia + Supabase (Postgres + Auth +
  RLS) with Playwright e2e. Applies e2e wait discipline (waitForTimeout
  as a flake suspect), Playwright trace/screenshot evidence, Pinia
  state isolation between tests, Supabase RLS-silently-returns-empty
  diagnosis, and generated-types drift after schema changes.

### Step 3 — Form a root-cause hypothesis

State an explicit, falsifiable hypothesis about **why** the failure
occurs — not just **what** failed (the assertion already says what).
Write it down in the transcript before designing an experiment. A good
hypothesis names a location and a mechanism: "the password-reset flow
rejects a valid token at the accept step because the token was signed
with a different secret than the verifier reads — the assertion trips
at token acceptance, but the defect is in the signing-config wiring
upstream."

### Step 4 — Design + run the cheapest discriminating experiment

Design the cheapest experiment whose outcome **differs depending on
whether the hypothesis is true**. A discriminating experiment is not a
retry; it is a probe:

- a **log / one-off print** of the suspect function's actual return at
  the boundary (e.g. `node -e` calling the module directly) — confirms
  whether the data or the render is wrong;
- a **narrower test invocation** — run just the unit under suspicion,
  then the next one up, to localize which layer flips the result;
- a **bisect over inputs** — feed the suspect function a minimal,
  hand-built input where the bug must show vs. an input where it
  cannot, and observe the difference;
- a **git bisect** over history when "this started failing after the
  merge" — let the binary search find the introducing commit.

Run it. Record the result in the transcript. **Confirm or refute.** If
refuted, form the next hypothesis and narrow — do not edit on a
refuted hypothesis. Every code edit in this skill must be grounded in a
**confirmed** hypothesis. Shotgun edits and speculative retries are the
thrash this skill exists to prevent.

### Step 5 — Classify the defect (the heart of the skill)

From hierovision/ai-framework, MIT.

More from hierovision/ai-framework