incident-response

Files (1)

• SKILL.md

SKILL.md (first lines)

# Incident Response

## Severity Levels

Assign a severity level immediately upon incident detection. Severity determines response urgency, communication cadence, and escalation path.

| Severity | Name | Description | Examples | Response Time | Update Cadence | Responders |
|----------|------|-------------|----------|---------------|----------------|------------|
| **P0** | Total Outage | Complete service unavailability. All users affected. Revenue-impacting. | Site completely down, data corruption, security breach with active exploitation | < 15 minutes | Every 15 minutes | All hands on deck, executive notification |
| **P1** | Major Degradation | Core functionality severely impaired. Large portion of users affected. | Payment processing broken, authentication failing for most users, major data pipeline stalled | < 30 minutes | Every 30 minutes | On-call engineer + team lead, stakeholder notification |
| **P2** | Partial Impact | Non-core functionality broken or core functionality degraded for a subset of users. | Search feature down, slow responses in one region, intermittent errors for some users | < 2 hours | Every 2 hours | On-call engineer |
| **P3** | Minor Issue | Cosmetic issues, minor bugs, or issues with workarounds available. | UI glitch, non-critical background job delayed, minor data inconsistency | Next business day | Daily (if ongoing) | Assigned engineer |

### Escalation Rules

- If a P2 is not resolved within 4 hours, escalate to P1.
- If a P1 is not resolved within 2 hours, escalate to P0.
- Any incident involving data breach or security compromise is automatically P0.
- When in doubt, over-classify. It is better to downgrade than to under-respond.

## Triage Procedure

When an incident is detected (alert, user report, or monitoring), follow this triage sequence:

### Step 1: Assess Impact

Answer these questions within the first 5 minutes:

- [ ] Who is affected? (All users, subset, internal only)
- [ ] What is broken? (Specific feature, entire service, data integrity)
- [ ] When did it start? (Check monitoring for onset time)
- [ ] Is it getting worse? (Error rate trending up, stable, or recovering)

### Step 2: Assign Severity

Based on the impact assessment, assign a severity level using the table above. Document the severity and reasoning.

### Step 3: Assemble the Response Team

| Severity | Who to Notify |
|----------|--------------|
| P0 | On-call engineer, engineering manager, VP Engineering, customer support lead, communications |
| P1 | On-call engineer, engineering manager, customer support lead |
| P2 | On-call engineer, relevant team lead |
| P3 | Assigned engineer (via ticket) |

### Step 4: Designate Roles

For P0 and P1 incidents, explicitly assign these roles:

| Role | Responsibility |
|------|---------------|
| **Incident Commander (IC)** | Coordinates response, makes decisions, delegates tasks. Does NOT debug. |
| **Technical Lead** | Leads investigation and mitigation. Communicates findings to IC. |
| **Communications Lead** | Drafts and sends status updates to stakeholders and customers. |
| **Scribe** | Documents timeline, actions taken, and decisions in real time. |

### Step 5: Open Incident Channel

Create a dedicated communication channel (Slack channel, incident bridge call):

- Name it clearly: `#incident-2024-03-15-payments-down`
- Pin the initial assessment and severity level.
- All discussion and decisions happen in this channel.

## Communication Templates

### Initial Alert

```
INCIDENT DECLARED: [P0/P1/P2] - [Brief description]

Impact: [Who is affected and how]
Start time: [When it began, UTC]
Current status: [Investigating / Identified / Mitigating]
Incident Commander: [Name]
Channel: #incident-[date]-[topic]

Next update in [15/30] minutes.
```

### Status Update

```
INCIDENT UPDATE: [P0/P1/P2] - [Brief description]

Status: [Investigating / Identified / Mitigating / Resolved]
Duration: [Time since start]
Current state: [What is happening now]
Actions taken: [What has been tried]
Next steps: [What will be done next]

Next update in [15/30] minutes.
```

### Resolution Notification

```
INCIDENT RESOLVED: [P0/P1/P2] - [Brief description]

Duration: [Total time from detection to resolution]
Root cause: [One-sentence summary]
Resolution: [What fixed it]
Customer impact: [Summary of user-facing impact]
Follow-up: Post-incident review scheduled for [date/time].

Monitoring for recurrence.
```

### Customer-Facing Communication

```
[Service Name] Status Update

We are aware of an issue affecting [description of user impact].
Our team is actively working to resolve this.

Started: [Time, timezone]
Current status: [Brief, non-technical description]

From Claude-Code-Community-Ireland/claude-code-resources, MIT.

More from Claude-Code-Community-Ireland/claude-code-resources