systematic-debugging

Files (11)

• CREATION-LOG.md
• SKILL.md
• condition-based-waiting-example.ts
• condition-based-waiting.md
• defense-in-depth.md
• find-polluter.sh
• root-cause-tracing.md
• test-academic.md
• test-pressure-1.md
• test-pressure-2.md
• test-pressure-3.md

SKILL.md (first lines)

# Systematic Debugging

## Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

**Violating the letter of this process is violating the spirit of debugging.**

## The Iron Law

```
NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST
```

## 🚨 Emergency Checklist (Senior Engineer)

**Production/High-Severity Incident? Read this first.**

1. **Stop the Bleeding**: Can we rollback? Toggle a feature flag? Restart? (Mitigate *then* debug).
2. **Snapshot State**: Capture logs, heap dumps, db state *before* restarting/fixing.
3. **Communication**: Who needs to know? (Status page, stakeholders). Don't debug in a silo.
4. **Bisect**: If a deploy caused it, bisect commits immediately.
5. **Hypothesis Log**: Write down what you've checked. "I think it's DB" -> Checked DB -> "It's not DB". Prevent circular debugging.

---

If you haven't completed Phase 1, you cannot propose fixes.

## When to Use

Use for ANY technical issue:

- Test failures
- Bugs in production
- Unexpected behavior
- Performance problems
- Build failures
- Integration issues

**Use this ESPECIALLY when:**

- Under time pressure (emergencies make guessing tempting)
- "Just one quick fix" seems obvious
- You've already tried multiple fixes
- Previous fix didn't work
- You don't fully understand the issue

**Don't skip when:**

- Issue seems simple (simple bugs have root causes too)
- You're in a hurry (rushing guarantees rework)
- Manager wants it fixed NOW (systematic is faster than thrashing)

## The Four Phases

You MUST complete each phase before proceeding to the next.

### Phase 1: Root Cause Investigation

**BEFORE attempting ANY fix:**

1. **Read Error Messages Carefully**
   - Don't skip past errors or warnings
   - They often contain the exact solution
   - Read stack traces completely
   - Note line numbers, file paths, error codes

2. **Reproduce Consistently (Script It!)**
   - **Senior Pattern:** Don't rely on "clicking around".
   - Create a `repro.py`, `repro.sh`, or `repro.js` script.
   - Script should: Setup state -> Trigger bug -> Assert failure.
   - *Why?* Allows rapid iteration of fixes and proves the fix works definitively.
   - If not reproducible locally → Iterate on logging in staging until you trap it.

3. **Check Recent Changes (Bisect)**
   - **Senior Pattern:** `git bisect` is your best friend.
   - If it worked yesterday and fails today, the answer IS in the diff.
   - Don't guess which file; let the binary search find it.
   - Check infrastructure-as-code diffs too, not just app code.

4. **Gather Evidence in Multi-Component Systems**

   **WHEN system has multiple components (CI → build → signing, API → service → database):**

   **BEFORE proposing fixes, add diagnostic instrumentation:**

   ```
   For EACH component boundary:
     - Log what data enters component
     - Log what data exits component
     - Verify environment/config propagation
     - Check state at each layer

   Run once to gather evidence showing WHERE it breaks
   THEN analyze evidence to identify failing component
   THEN investigate that specific component
   ```

   **Example (multi-layer system):**

   ```bash
   # Layer 1: Workflow
   echo "=== Secrets available in workflow: ==="
   echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

   # Layer 2: Build script
   echo "=== Env vars in build script: ==="
   env | grep IDENTITY || echo "IDENTITY not in environment"

   # Layer 3: Signing script
   echo "=== Keychain state: ==="
   security list-keychains
   security find-identity -v

   # Layer 4: Actual signing
   codesign --sign "$IDENTITY" --verbose=4 "$APP"
   ```

From whd4/expert-ai-skills, MIT.

More from whd4/expert-ai-skills