reliability

Files (5)

• SKILL.md
• common-mistakes.md · anti-patterns/
• incident-checklist.md · checklists/
• release-readiness.md
• output-validation.md · validation/

SKILL.md (first lines)

# Nexus Production Incident Investigator

Time-boxed, evidence-driven workflow for investigating production incidents and evaluating release readiness.

---

## Compatibility
- Sub-skills: `release-readiness.md` | Materials: `checklists/incident-checklist.md`, `anti-patterns/common-mistakes.md`, `validation/output-validation.md`
- Required: Read, Bash, Grep | Optional: WebSearch | Hands off to: `nexus:debugging` (code RCA), the `roadmap-planner` agent (post-mortem actions)

---

## Core Principle

**Stabilize first. Investigate second. Never skip the timeline.**

---

## Incident Classification

| Severity | Definition | Response target |
|----------|-----------|-----------------|
| **P0** | Complete outage or data loss affecting all/most users | Immediate; page all on-call |
| **P1** | Major feature broken or >25% requests failing | < 5 min acknowledgement; page on-call lead |
| **P2** | Degraded performance or partial failure | < 15 min acknowledgement; one engineer |
| **P3** | Minor anomaly, no user impact | Next business hour |

If you cannot classify immediately, default to P1 and downgrade after Phase 1.

---

## Investigation Workflow

### Phase 1 — Triage (0–5 min)

1. **Confirm the signal** — cross-check alert against at least one independent source (logs, dashboard, manual repro). Never declare from a single data point.
2. **Classify severity** using the table above.
3. **Identify blast radius** — which services, user segments, and whether data integrity is at risk.
4. **Open incident channel** and post: `[P0/P1/P2] <service> — <symptom> — investigating`
5. **Assign roles** (P0/P1): Incident Commander (comms only), Tech Lead (investigation), Scribe (timeline).

---

### Phase 2 — Stabilize (5–15 min)

**Apply mitigations before investigating.** Stop at first "yes":

| Question | Action |
|----------|--------|
| Deploy in last 2 hours? | Roll back; confirm error rate drops |
| Config/feature-flag change? | Revert; confirm error rate drops |
| One bad instance/pod? | Remove from LB, restart |
| Downstream dependency down? | Enable fallback/circuit breaker |
| DB connection pool exhausted? | Restart pooler, reduce pool size |
| Traffic spike causing saturation? | Rate limit, add capacity, shed load |

After mitigation, post: `[Update] Mitigation: <what>. Error rate: <before> → <after>. Investigating root cause.`

---

### Phase 3 — Root Cause (15–60 min)

#### 3.1 — Build the Timeline

Start here. Every other step depends on this. Format: `T-Xh: [event]` through `T+Xm: [resolution]`. Sources: deployment history, config audit logs, monitoring dashboards, application logs, infrastructure events (CloudTrail, K8s events).

#### 3.2 — Signal Analysis

Check in priority order:

| Metric | What it tells you |
|--------|-------------------|
| Error rate (5xx/4xx) | Which endpoints are failing |
| Latency p99 | Slowness vs. failure distinction |
| Saturation (CPU, memory, connections) | Resource exhaustion |
| Traffic volume (RPS) | Spike-driven vs. change-driven |
| Queue depth / backlog | Upstream pressure |
| Database metrics | Connection count, lock waits, slow queries |
| Dependency health | Which downstreams are responding |

**USE Method:** Utilization → Saturation → Errors. Apply to CPUs, memory, disk I/O, network, connection pools.

#### 3.3 — Correlation (priority order)

1. Deploys (within 2h) 2. Config/flag changes 3. Infrastructure changes 4. Traffic spikes 5. Dependency changes 6. Time-based patterns (cron/batch) 7. Data patterns (user/tenant/shape)

**Rule: 80% of incidents are caused by the most recent deploy or config change.**

```bash
git log --oneline --after="2 hours ago"
kubectl rollout history deployment/<service>
aws cloudtrail lookup-events --start-time ...
```

#### 3.4 — Hypothesis Testing

For each hypothesis: what evidence supports it, what would disprove it, does it explain the full timeline? Root cause is confirmed when: (1) it explains all symptoms, (2) it explains why the mitigation worked, (3) you can state it in one sentence with a mechanism. If you cannot — you have a symptom, not a root cause.

#### 3.5 — Reproduce in Staging (P1/P2)

Reproduce in staging, confirm symptoms match production, confirm fix resolves it before pushing.

---

### Phase 4 — Post-Mortem (within 48h of resolution)

See `checklists/incident-checklist.md` for the full checklist. Required sections: Timeline, Root Cause (one sentence), Why It Happened, Why We Didn't Catch It Earlier, Mitigation/Resolution, Action Items (owner + due date + type), What Went Well, What Went Poorly.

---

## Release Readiness

Route "is this safe to deploy" questions to `release-readiness.md` (pre-deploy checklist, risk scoring, Go/No-Go).

---

## Output Contract

Every investigation must produce this report:

From aayushostwal/nexus, MIT.

More from aayushostwal/nexus