go-troubleshooting

Files (4)

• SKILL.md
• common-bugs.md · references/
• flaky-and-build.md · references/
• methodology.md · references/

SKILL.md (first lines)

# Go Troubleshooting

Diagnosis before fixes. Reproduce, observe, localize, then change code.
Never "fix" a symptom you haven't explained — the bug will move.

Detailed reference material, loaded on demand:

- `references/methodology.md` — the debugging loop, tracing callers before
  blaming code, `git bisect`, red flags in your own reasoning.
- `references/flaky-and-build.md` — reproducing flaky tests and their usual
  causes; build, module and toolchain errors with their fixes.
- `references/common-bugs.md` — symptom → bug → fix tables for the Go bugs
  that compile cleanly and misbehave.

Read a reference file only when the summary below is not enough.

## 0. Method

1. State expected vs actual, and collect the full evidence (whole stack,
   exact input, trace ID).
2. Reproduce it in a failing test, then shrink the reproduction.
3. One hypothesis, one change, one observation — never stacked changes.
4. Fix the cause, not the trigger: keep asking why the bad value existed.
5. Three failed fixes means the mental model is wrong — start over from
   the evidence. If the cause predates you, `git bisect run` finds the
   commit.

## 1. Pick the Procedure by Symptom

| Symptom | Procedure |
|---|---|
| Wrong result, no crash | §0 Method, then `references/common-bugs.md` |
| Crash with stack trace | §2 Read the panic |
| Program hangs / requests stall | §3 Dump goroutines, find the block |
| `fatal error: all goroutines are asleep` | §3 — Go detected total deadlock |
| Memory grows until OOM | §4 Heap profile diff |
| Goroutine count grows | §5 Goroutine profile diff |
| Intermittent corrupt data / weird values | §6 Race detector |
| Need to inspect state interactively | §7 Delve |
| Incident in production right now | §11 Capture before restart |
| Test fails intermittently | §9 Flaky tests |
| Build, module or toolchain error | §10 Build failures |

## 2. Reading a Panic

```text
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6bb0e4]

goroutine 43 [running]:
myapp/internal/service.(*UserService).Notify(0x0, {0xc000123456?, ...})
        /app/internal/service/user.go:87 +0x24
myapp/internal/handler.(*Handler).Create(0xc0001a2000, ...)
        /app/internal/handler/user.go:41 +0x1c5
```

Read it mechanically:

1. First line: what kind of panic. `nil pointer dereference` + `addr=0x0`
   means a nil receiver, nil field, or nil map/pointer argument.
2. Top frame in YOUR code: `user.go:87` — go there.
3. Receiver value in the frame: `(*UserService).Notify(0x0, ...)` — the
   `0x0` first argument IS the receiver: the service itself was nil.
   Trace where it was constructed (or wasn't).
4. `goroutine 43` — if it's not goroutine 1, find who spawned it and
   whether a recover boundary should exist there.

## 3. Hangs and Deadlocks

Get a goroutine dump from the hanging process:

```bash
kill -QUIT <pid>      # dumps all goroutine stacks to stderr, then exits
# or, if net/http/pprof is mounted:
curl 'localhost:6060/debug/pprof/goroutine?debug=2'
```

Then classify the stacks:

- `[semacquire]` on `sync.(*Mutex).Lock` — find which goroutine HOLDS
  the mutex: look for another stack inside the critical section. Two
  goroutines each holding one of two locks = lock-order inversion.
- `[chan send]` / `[chan receive]` — the other side is gone. Find who
  should be receiving/sending and why it exited (or was never started).
- `[select]` with a `ctx.Done()` case missing — blocked call that
  ignores cancellation.
- Hundreds of identical stacks — that's a leak (§5), not a deadlock.

Slow rather than stuck? Enable `runtime.SetMutexProfileFraction` /
`SetBlockProfileRate` and read the mutex and block profiles, or take a
5-second trace (`/debug/pprof/trace?seconds=5`) to see goroutines that are
runnable but not running. `kill -QUIT` ends the process — in production,
prefer the pprof endpoint.

## 4. Memory Leaks

Mount `net/http/pprof` in long-running services on a private listener
(`localhost:6060`), never on the public mux.

Diff heap profiles over time — a leak is growth that never returns:

```bash
curl -s localhost:6060/debug/pprof/heap > heap1.pb.gz
sleep 300   # let the leak accumulate
curl -s localhost:6060/debug/pprof/heap > heap2.pb.gz
go tool pprof -base heap1.pb.gz heap2.pb.gz
(pprof) top          # biggest positive delta = the leak
(pprof) list FuncName
```

Usual suspects: unbounded caches/maps without eviction, subslices
pinning large arrays, `time.Ticker` never stopped, response bodies not
closed, growing global slices, forgotten goroutines holding buffers.

## 5. Goroutine Leaks

On Go 1.27+, start with the leak profile — goroutines blocked on a channel
or lock that nothing can ever release:

From eduardo-sl/go-agent-skills, MIT.

More from eduardo-sl/go-agent-skills