release-it

Files (7)

• SKILL.md
• anti-patterns.md · references/
• capacity-planning.md · references/
• chaos-engineering.md · references/
• deployment-strategies.md · references/
• observability.md · references/
• stability-patterns.md · references/

SKILL.md (first lines)

# Release It! Framework

Framework for designing, deploying, and operating production-ready software. The software that passes QA is not the software that survives production — production is hostile, and systems must expect and handle failure at every level.

## Core Principle

**Every system will eventually be pushed beyond its design limits.** The question is not whether failures happen, but whether your system degrades gracefully or collapses catastrophically. Production-ready software is not just correct — it is resilient, observable, and operates through partial failures without human intervention.

## Scoring

**Goal: 8/8.** Score a production system by the Quick Diagnostic: **1 point per row answered "yes"** across the 8 checks (timeouts, circuit breakers, bulkheads, zero-downtime deploy, deep health checks, correlated telemetry, load-tested past peak, failure injection). Bands: **7-8** = every integration point is bounded, isolated, observable, and deploy/release are decoupled; **4-5** = some patterns present but ≥3 diagnostic rows fail (e.g. unbounded retries, shared pools, shallow health checks); **≤2** = relies on the happy path with no breakers, no capacity model, no failure testing. Always state the current score, the failing rows, and the specific fix for each.

## The Release It! Framework

Six areas that determine whether software survives contact with production:

### 1. Stability Anti-Patterns

**Core concept:** Failures propagate through integration points and cascade across system boundaries. The most dangerous patterns are not bugs in your code — they are emergent behaviors when systems interact under stress.

**Why it works:** These patterns recur across outages, so audit by name: walk every integration point and ask which anti-pattern it currently enables, then close that specific crack rather than hardening at random.

**Key insights:**
- Integration points are the number-one killer — every socket, HTTP call, or queue is a risk
- Slow responses are worse than no response: they tie up threads, exhaust pools, and propagate delay up the call chain
- Unbounded result sets turn a harmless query into an out-of-memory crash once data outgrows test assumptions
- Users generate load no test predicts — bots, retry storms, flash crowds; self-denial attacks happen when your own marketing overwhelms your infrastructure
- Blocked threads are the silent killer — deadlocks and contention show no errors until everything stops

**Code applications:**

| Context | Guard | Example |
|---------|-------|---------|
| HTTP calls | Assume every remote call can fail, hang, or return garbage | Wrap all external calls with timeout + circuit breaker |
| Database queries | Enforce result set limits | Add `LIMIT`; paginate all list endpoints |
| Thread pools | Isolate pools per dependency | Separate pool for payment gateway vs. search |
| Marketing events | Coordinate launches with capacity planning | Pre-scale before Black Friday; queue coupon redemptions |

See [references/anti-patterns.md](references/anti-patterns.md) when triaging an outage or hardening an integration point — each anti-pattern with its failure scenario and the symptom that detects it.

### 2. Stability Patterns

**Core concept:** Counter each anti-pattern with a stability pattern: circuit breakers stop cascades, bulkheads isolate blast radius, timeouts reclaim stuck resources. Together they make a system bend under load instead of breaking.

**Why it works:** Each pattern caps the damage one failure can do: a breaker trip converts an unbounded cascade into a fast local rejection, a bulkhead confines the outage to one pool. Treat a tripped breaker as expected output, not an incident — page on the breaker *staying* open, not on it opening.

**Key insights:**
- Circuit Breaker: three states (closed, open, half-open) — trips after threshold failures, periodically tests recovery
- Timeouts: every outbound call needs connect AND read timeouts, propagated up the call chain
- Retry with exponential backoff + jitter prevents thundering herd on recovery
- Fail Fast: reject requests you know will fail instead of wasting resources; Handshaking lets the server decline work before it's sent
- Steady State: systems accumulate cruft (logs, sessions, temp files) — design automatic cleanup
- Let It Crash: a clean restart often beats limping along in an unknown state

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| Service calls | Circuit Breaker | Open after 5 failures in 60s; half-open after 30s |
| Resource isolation | Bulkhead | Dedicated connection pools for critical vs. non-critical |
| Network calls | Timeout with propagation | Connect 1s, read 5s; propagate deadline downstream |
| Retries | Backoff + jitter + budget | Base 100ms, max 3 retries, 20% fleet retry budget |
| Data cleanup | Steady State | Purge sessions >24h; rotate logs at 500MB |

See [references/stability-patterns.md](references/stability-patterns.md) when implementing a breaker or tuning thresholds — state-machine diagram, parameter ranges, what-counts-as-failure tables, and how to combine patterns.

### 3. Capacity and Availability

**Core concept:** Capacity is not one number — it is a multi-dimensional function of CPU, memory, network, disk I/O, connection pools, and threads. Capacity planning means knowing which resource bottlenecks first, and at what load.

**Why it works:** Untested systems fail at peak load — the worst possible moment. Knowing actual (not theoretical) limits lets you set realistic SLAs and scale before users hit the wall.

**Key insights:**
- Test taxonomy: load test (expected traffic), stress test (beyond limits), soak test (sustained, catches leaks), spike test (sudden bursts)
- Universal Scalability Law: throughput never scales linearly — contention and coherence costs cause diminishing returns
- Pool exhaustion looks identical to a database outage from the application's perspective; size pools from measured concurrency, not defaults
- "The cloud is infinitely scalable" is a myth — auto-scaling has lag, cold starts, and hard limits

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| Load testing | Ramp to peak, then 2x, observe degradation | Increase RPS until latency exceeds SLO |
| Connection pools | Size from measured concurrency | Set pool to P99 active connections + 20% headroom |
| Soak testing | 80% capacity for 24-72 hours | Catch memory/connection/file-handle leaks |
| Capacity model | Document bottleneck per service | "Service X is memory-bound at 2000 RPS; 4GB per instance" |

See [references/capacity-planning.md](references/capacity-planning.md) when planning a load test or sizing pools — test methodologies, pool/thread tuning, and Universal Scalability Law modeling.

### 4. Deployment and Release

**Core concept:** Deployment (putting code on servers) and release (exposing it to users) are separate operations that should be decoupled — deploy without risk, release with confidence.

**Why it works:** Most outages are caused by changes. Decoupling lets you deploy to production, verify, and only then route traffic; if something breaks, you roll back the release, not the deployment.

**Key insights:**
- Zero-downtime deployment is non-negotiable: rolling, blue-green, or canary
- Feature flags dark-launch code and enable it independently of deployment
- Database migrations must be backward-compatible — old and new code run simultaneously during deploys (expand-contract)
- Immutable infrastructure: never patch a running server — build a new image, deploy, destroy the old
- Rollback must be faster than roll-forward; if rollback takes 30 minutes, you will avoid deploying

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| Deploys | Blue-green with health check gate | Deploy to green; smoke test; swap router |
| Progressive rollout | Canary with automated rollback | 5% traffic to canary; auto-rollback if error rate >1% |
| Feature launch | Flags with emergency off switch | Ship behind flag; enable for 10%; monitor; ramp |
| Schema changes | Expand-contract migration | Add column; write both; backfill; drop old |

See [references/deployment-strategies.md](references/deployment-strategies.md) when planning a release or a schema change — blue-green/canary/rolling mechanics, expand-contract migration steps, and infrastructure-as-code.

### 5. Health Checks and Observability

**Core concept:** You cannot operate what you cannot observe. Health checks, metrics, logs, and traces are the sensory organs of your system in production — a first-class design concern, not an afterthought.

**Why it works:** Untraced failures are invisible until a user reports them. Emit high-cardinality, structured events (not just pre-aggregated counters) so you can ask new questions of past incidents without shipping new instrumentation first.

From wondelai/skills, MIT.

More from wondelai/skills