runbook

Files (2)

• SKILL.md
• incident-template.md · references/

SKILL.md (first lines)

# Runbook

Structured incident investigation: symptom intake, triage, investigation, resolution, documentation. Every step collects evidence.

**Core principle:** Collect evidence before forming hypotheses. Do not fix what you have not diagnosed.

## When to Use

- Production alert fires and needs investigation
- Service outage or degraded performance
- Error rate spike or new error pattern
- User-reported issue with unclear cause
- Post-deploy regression

**Not for:** planned maintenance, feature development, or issues with obvious root cause already known.

## Process

```
1. Intake    — Capture symptoms, timeline, blast radius
2. Triage    — Severity assessment, who to notify, immediate mitigation
3. Investigate — Systematic evidence gathering, hypothesis testing
4. Resolve   — Apply fix, verify, monitor
5. Document  — Post-incident report with timeline and learnings
```

### 1. Intake

Capture before investigating:

- **What** — exact symptoms, error messages, affected endpoints
- **When** — first occurrence, pattern (constant, intermittent, escalating)
- **Where** — which services, environments, regions
- **Who** — affected users/systems, scope estimate
- **What changed** — recent deploys, config changes, infrastructure changes

```bash
# Quick checks to establish timeline
git log --oneline --since="2 hours ago"   # recent deploys
# Check monitoring dashboards, alert history
# Check infrastructure change logs
```

### 2. Triage

| Signal | Severity | Response |
|--------|----------|----------|
| Data loss, security breach | Critical | All hands, immediate mitigation, notify stakeholders |
| Service fully down | High | On-call team, rollback consideration |
| Degraded performance, partial outage | Medium | Investigate, communicate ETA |
| Cosmetic, non-blocking | Low | Queue for next business day |

**Immediate mitigation options** (before root cause is known):
- Rollback last deploy
- Scale up / restart affected services
- Enable feature flag / circuit breaker
- Redirect traffic away from affected component

### 3. Investigate

Systematic, not ad hoc. Work from evidence, not hunches.

**Evidence gathering order:**
1. Logs around incident start time (correlate across services)
2. Metrics — error rates, latency, resource usage, queue depths
3. Recent changes — deploys, config, infrastructure
4. Dependencies — upstream/downstream service health
5. Reproduce — can you trigger the issue in a controlled way?

**Hypothesis testing:** Form hypothesis, predict what evidence would confirm or refute, check that evidence. Do not confirmation-bias toward the first theory.

Use arsyn:debugging for deep root cause analysis when the surface investigation identifies the failing component but not the underlying cause.

### 4. Resolve

- Apply targeted fix (not a workaround unless time-critical)
- Verify fix in production — monitor for 15-30 min
- Confirm affected metrics return to baseline
- Remove any temporary mitigations applied in triage

### 5. Document

Write post-incident report. See references/incident-template.md for format.

Key sections: timeline, root cause, impact, resolution, action items. The report serves future investigations — be specific about what worked and what did not during diagnosis.

## Quick Reference

| Phase | Key question | Output |
|-------|-------------|--------|
| Intake | What exactly is happening? | Symptom summary with timeline |
| Triage | How bad is it? | Severity level + immediate actions |
| Investigate | Why is it happening? | Root cause with evidence |
| Resolve | Is it fixed? | Verified fix + monitoring |
| Document | What did we learn? | Post-incident report |

## Integration

- **arsyn:debugging** — deep root cause tracing when surface investigation is not enough
- **arsyn:verifying** — verify fix meets requirements before closing
- **arsyn:precision** — careful execution when applying production fixes

## Gotchas

- **Fixing before diagnosing** — Applying a "likely fix" without confirming root cause often masks the real issue or introduces new ones. Collect evidence first.
- **Tunnel vision on first hypothesis** — The most obvious explanation is not always correct. Actively seek disconfirming evidence.
- **Skipping documentation** — "We'll write it up later" means it never gets written. Capture timeline during investigation, not after.
- **Incomplete blast radius assessment** — Check downstream consumers, async jobs, and cache layers. Visible symptoms are often a subset of actual impact.
- **Premature all-clear** — Monitor for at least 15 minutes after fix. Some issues resurface under load or on the next cron cycle.
- **Not checking what changed** — Most production incidents correlate with a recent change. Always check deploy history and config changes early.

From renathoaz/arsyn, MIT.

More from renathoaz/arsyn