incident-response

Files (2)

• SKILL.md
• incident-playbook.md · references/

SKILL.md (first lines)

# Incident Response

Manage active production incidents from detection to resolution. Stack-agnostic. Tool-agnostic.

This skill is for active incidents and incident process. For after-the-fact analysis, use `after-action-report`. For planned launches, use `launch-runbook`.

---

## When to use

- An active incident is happening
- Building incident response procedures
- Defining severity levels
- Setting up on-call rotations
- Training a team on incident response

## When NOT to use

- Post-incident retrospective (use `after-action-report`)
- Planned launches (use `launch-runbook`)
- Pre-launch issue triage (use `qa-testing`)

---

## Required inputs

- Awareness of the incident (alert, customer report, internal observation)
- Access to production systems and monitoring
- Roles and authorities clearly defined
- Communication channels operational

---

## The framework: 5 phases

### 1. Detection

How the incident becomes known.

**Detection sources:**

- Automated alerts (monitoring, SLO violations, error rate spikes)
- Customer reports (support tickets, social media, status page subscribers)
- Internal observation (engineer notices something off)
- Third-party (security researchers, partners)

**On detection:**

- Acknowledge within target time (typically 5 to 15 minutes for critical)
- Assess severity (see severity rubric below)
- Page the on-call if not already paged
- Open the incident channel

### 2. Triage

Establish severity and impact.

**Severity rubric:**

| Severity | Definition | Response |
|---|---|---|
| SEV-1 (Critical) | Major customer-facing functionality broken. Data integrity at risk. Security breach. | All-hands. Incident commander. Active war room. Public communication required. |
| SEV-2 (Major) | Significant degradation. Some customers affected. Revenue impact. | Incident commander assigned. Active response. Internal communication. May or may not need public communication. |
| SEV-3 (Minor) | Limited impact. Workaround available. Affecting a small group of users. | Standard on-call response. Single owner. |
| SEV-4 (Low) | Cosmetic, edge-case, or low-frequency. No urgent action needed. | Tracked as bug. Addressed in normal queue. |

Severity can change. Re-evaluate as more info emerges.

### 3. Mitigation

Stop the bleeding before fixing the cause.

**Mitigation patterns (faster than full fix):**

- **Rollback** (revert recent deploy)
- **Feature flag off** (disable the broken feature without deploy)
- **Failover** (route to healthy replica or region)
- **Scale up** (more capacity to absorb the load)
- **Throttle** (reject some traffic to protect the rest)
- **Graceful degradation** (turn off non-essential features to keep core functional)
- **Maintenance mode** (last resort, blocks all users)

**Mitigation principle:** Stop user impact first. Cause analysis second.

### 4. Communication

Three audiences during an incident:

**Internal team:**
- Real-time updates in incident channel
- Cadence: every 15 minutes minimum during active incident
- Format: timestamped status updates with what we know, what we're doing, ETA

**Internal stakeholders:**
- Higher-level updates to broader org
- Cadence: every 30 to 60 minutes
- Format: business-impact framing, not technical detail

**External / customers:**
- Status page updates
- Cadence: every 30 minutes minimum during active incident
- Format: plain language, no blame, what users are experiencing, what to expect

**Communication principles:**
- Acknowledge before you have answers ("We're aware and investigating")
- Update on schedule even if no progress ("Still investigating, no new information")
- Never speculate publicly about cause
- Confirm resolution explicitly when restored

### 5. Resolution

Verified fix, customers restored, incident closed.

**Resolution criteria:**

- Mitigation in place and verified
- Root cause identified (or explicitly deferred to AAR)
- All affected systems back to normal
- Customers can resume normal use

From rampstackco/claude-skills, MIT.

More from rampstackco/claude-skills