sre

Files (1)

• SKILL.md

SKILL.md (first lines)

# Site Reliability Engineering

## Iron Law

```
70% of outages are caused by changes, make every change incremental, observable, and reversible.
Alert on symptoms, not causes, every page that does not require immediate human action is a bug.
MTTR beats MTBF, optimise for fast recovery, not for preventing all failures.
```

---

## Before Taking Any Action

1. **Announce** what you intend to produce, SLO proposal, alert rules, runbook, IaC, postmortem, PRR report, chaos experiment design
2. **Confirm scope**, which service, which environment, existing SLOs if any
3. **Ask for confirmation** before writing files, running commands, or proposing infrastructure changes
4. **Report** what was produced, what decisions were embedded, and what still needs human review (SLO targets need stakeholder sign-off; chaos experiments need a maintenance window)

---

## Task Approach

Use this table to determine what to produce for each task type:

| User asks for | What to produce |
|---|---|
| SLO / SLI definition | SLI selection per service type (see framework below) + SLO target + error budget calculation + error budget policy table |
| Observability design | Four Golden Signals instrument checklist + Prometheus metrics + Grafana dashboard JSON or dashboard-as-code spec + alert rules |
| Alert rules | Burn rate alerts (fast + slow) per SLO using the two-alert model; include runbook link placeholders |
| Production readiness review | PRR report structured against the six-dimension checklist; flag each dimension as Pass / At Risk / Fail with remediation steps |
| Runbook | Step-by-step response procedure for a specific alert or failure mode; include detection, diagnosis steps, mitigation commands, escalation path, and rollback |
| Postmortem | Completed postmortem document using the blameless template; ask for the incident timeline and impact before writing |
| Toil identification | Toil audit: list recurring manual tasks, classify each against the toil criteria, propose automation approach and elimination priority |
| Infrastructure as code | Terraform modules or Kubernetes manifests following the IaC principles below; always include rollback plan as a comment block |
| Chaos experiment design | Hypothesis statement + blast radius definition + success/failure criteria + kill switch + measurement plan; do not generate commands that inject failure without explicit user confirmation |

---

## SLO/SLI/Error Budget Framework

### Definitions

| Term | Definition |
|---|---|
| **SLI** | A quantitative measure of service behaviour (e.g. % of requests with latency < 200ms) |
| **SLO** | A target for the SLI over a time window (e.g. 99.9% of requests < 200ms over 30 days) |
| **Error Budget** | `1 − SLO target`, the allowed failure rate (e.g. 0.1% of requests may fail the SLI) |
| **SLA** | An SLO with contractual consequences (penalties, credits) |

### SLI Selection by Service Type

| Service type | Recommended SLIs |
|---|---|
| Request/response (APIs) | % of requests with latency < threshold (p99); % of successful responses (non-5xx) |
| Batch / pipeline | Throughput (records/s); end-to-end latency; % of batches completing within SLA |
| Storage | Durability (data loss rate); read latency p99; write availability |
| Scheduled jobs | % of jobs completing within time budget; error rate |

**Use percentiles (p99, p95), never averages.** An average hides the tail, users on p99 latency are real users having a bad experience.

### Error Budget Calculation

```
Error Budget = (1 − SLO%) × Time Window
Example: 99.9% SLO over 30 days
Error Budget = 0.1% × 30 × 24 × 60 = 43.2 minutes
```

### Error Budget Policy

| Budget remaining | Action |
|---|---|
| > 50% | Full velocity; deploy freely |
| 25–50% | Normal operations; monitor burn rate |
| 10–25% | Elevated caution; no risky deployments without SRE review |
| < 10% | Reliability sprint; feature freeze; postmortem required for any incident |
| 0% exhausted | Feature freeze until budget resets; escalate to engineering leadership |

---

## Four Golden Signals

Build dashboards starting with these four. If all four are healthy, the service is probably healthy.

| Signal | What it measures | Alert condition |
|---|---|---|
| **Latency** | Response time for successful and failed requests | p99 > SLO threshold for > 5 minutes |
| **Traffic** | Request rate (RPS, events/s, QPS) | Anomalous spike or drop from baseline |
| **Errors** | Rate of failed requests (4xx client, 5xx server) | Error rate > SLO-derived burn rate |
| **Saturation** | CPU, memory, disk, connection pool utilisation | > 80% sustained for > 10 minutes |

Alert on symptoms (p99 latency exceeds SLO threshold), not causes (CPU > 80%). Causes are investigated; symptoms are paged.

---

## Alert Design Rules

- **Every page requires immediate human action.** A page that does not is a bug, it trains engineers to ignore alerts.
- **Alert on burn rate, not absolute values.** A 1-hour burn rate of 14.4× will exhaust a 30-day 99.9% budget in 5 days.
- **Two-alert model per SLO:**
  - Fast burn: 6-hour window, 5% budget consumed → page immediately
  - Slow burn: 3-day window, 10% budget consumed → ticket/warning
- **Alert fatigue is a failure mode.** Any alert firing > 5 times per week without action is a toil candidate.
- **Runbook link in every alert.** Engineers under pressure must not have to remember what to do.

---

## Production Readiness Review (PRR) Checklist

Run before a service goes to production or receives on-call coverage:

| Dimension | Pass criteria |
|---|---|
| **System architecture** | Dependencies documented; failure modes identified; no unmitigated single points of failure |
| **SLI instrumentation** | Four Golden Signals instrumented; dashboards deployed; alerts configured and tested |
| **Emergency response** | Runbook written and reviewed; incident response flow practiced by at least one engineer |
| **Capacity planning** | Load tested to 120% of expected peak; N+2 provisioned |
| **Change management** | Canary or blue/green configured; rollback tested and documented; deployment < 30 minutes |

From pranav8494/team-of-agents, MIT.

More from pranav8494/team-of-agents