debugging-and-error-recovery

Files (1)

• SKILL.md

SKILL.md (first lines)

# Debugging and Error Recovery

## Overview

Systematic debugging with structured triage. When something breaks, stop adding features, preserve evidence, and follow a structured process to find and fix the root cause. Guessing wastes time. The triage checklist works across this polyglot repo — Python (`uv` workspace), Go modules (go 1.24), the TS portal — for test failures, build errors, runtime bugs, and production incidents.

## When to Use

- A toolchain gate fails after a change (`uv ... pytest`, `pyright`, `ruff`, `go test -race`, `pnpm test`)
- A build breaks (`go build ./...`, a Python import error, `pnpm typecheck`)
- Runtime behavior doesn't match expectations (retrieval loop, lobe/path routing, ingestion, write-back)
- A bug report arrives
- An error appears in logs or gate output
- Something worked before and stopped working

## The Stop-the-Line Rule

When anything unexpected happens:

```
1. STOP adding features or making changes
2. PRESERVE evidence (error output, logs, repro steps)
3. DIAGNOSE using the triage checklist
4. FIX the root cause
5. GUARD against recurrence
6. RESUME only after verification passes
```

**Don't push past a failing test or broken build to work on the next feature.** Errors compound. A bug in Step 3 that goes unfixed makes Steps 4-6 wrong.

## The Triage Checklist

Work through these steps in order. Do not skip steps.

### Step 1: Reproduce

Make the failure happen reliably. If you can't reproduce it, you can't fix it with confidence.

```
Can you reproduce the failure?
├── YES → Proceed to Step 2
└── NO
    ├── Gather more context (logs, environment details)
    ├── Try reproducing in a minimal environment
    └── If truly non-reproducible, document conditions and monitor
```

**When a bug is non-reproducible:**

```
Cannot reproduce on demand:
├── Timing-dependent?
│   ├── Add timestamps to logs around the suspected area
│   ├── For Go workers, run `go test -race ./...` to widen the race window
│   └── Run under load / concurrency to increase collision probability
├── Environment-dependent?
│   ├── Compare toolchain versions (Python via uv, Go pinned to 1.24, pnpm/Node for the portal)
│   ├── Check for differences in data (empty vs populated pgvector KB)
│   └── Confirm TEST_DATABASE_URL + pgvector are present — DB-dependent tests skip silently otherwise
├── State-dependent?
│   ├── Check for leaked tenant state between tests (every row/query/cache key carries tenant_id)
│   ├── Look for module-level globals, singletons, or shared pools
│   └── Run the failing scenario in isolation vs after other operations
└── Truly random?
    ├── Add defensive logging at the suspected location
    ├── Go: run the race detector (`go test -race ./...`)
    └── Document the conditions observed and revisit when it recurs
```

For test failures, run the **owning toolchain's** gate:
```bash
# Python member (uv workspace) — a single failing test, verbose
uv --directory packages/rag-core run python -m pytest -q -k TestName
uv --directory packages/rag-core run python -m pytest -vv path/to/test_file.py::test_name

# Go worker module — race detector for suspected concurrency bugs
go test -race -run TestName ./...

# TS portal
pnpm --filter portal test -- -t "test name"
```

Remember: **DB-dependent tests skip unless `TEST_DATABASE_URL` (with pgvector) is available.** A "passing" run that actually skipped the relevant tests is a false green. Start a `pgvector/pgvector:pg16` container, run `alembic upgrade head`, and export the DSN before trusting a green run.

### Step 2: Localize

Narrow down WHERE the failure happens:

```
Which layer is failing?
├── Bot runtime / interpreter → packages/{rag-core,agent-core,arag-core}: the generic
│                                interpreter over BotPolicy (synthesize/cite/filter stages)
├── Retrieval / KB tool        → retrieve_kb / packages/rag-core: ACL, ranking, compression,
│                                citation grounding
├── Ingestion                  → packages/ingest-core, apps/worker-ingest: chunking, embedding
├── LLM transport              → packages/llm-transport: anthropic-sdk-python → MiniMax endpoint,
│                                temperature, retries, streaming
├── API surface                → apps/backend (FastAPI), OpenAPI contract, portal type drift
├── Database / vectors         → Postgres + pgvector, alembic migrations, schema/indexes
├── Go workers                 → apps/{worker-mello,worker-mezon-go}, packages/*-sdk (go 1.24)
├── Portal                     → apps/portal (React + Vite); check generated OpenAPI types
├── MCP boundary               → kb.* tool contract, embedded transport
└── Test itself                → Is the test correct? Did it actually run, or skip on a missing DB?
```

**Use bisection for regression bugs:**
```bash
git bisect start
git bisect bad                    # Current commit is broken
git bisect good <known-good-sha>  # This commit worked
# Git checks out midpoints; run the owning toolchain's gate at each
git bisect run uv --directory packages/rag-core run python -m pytest -q -k TestName
```

### Step 3: Reduce

Create the minimal failing case:

- Remove unrelated code/config until only the bug remains

From minhlucvan/emdesign, Apache-2.0.

More from minhlucvan/emdesign