risk-based-testing

Files (2)

• SKILL.md
• examples.md · references/

SKILL.md (first lines)

<objective>
Equal coverage across all features wastes effort on low-risk areas while leaving critical paths under-tested — a 90% coverage target on a settings page is effort stolen from checkout. This skill discovers risk, quantifies it as impact × probability, maps test density to risk zones, and keeps the assessment current as the product evolves. Output is a scored risk matrix that feeds `test-strategy` and `test-planning`.
</objective>

## Quick Route

| Situation | Start at |
|-----------|----------|
| New product, no risk model yet | Phase 1 (Identification) → run the full 6 phases |
| Post-incident reassessment | Phase 6 (Reassessment triggers), then re-score the affected items in Phase 2 |
| AI/LLM feature to assess | Phase 3 (AI/LLM failure classes), score each class in Phase 2 |
| Sprint refresh of an existing matrix | Phases 4–5 (Heatmap + Coverage alignment) on changed features |
| Verify an old heatmap is still true | Phase 6 signals + Anti-Pattern "Risk Theater" |

---

## Discovery Questions

Check `.agents/qa-project-context.md` first — if it exists, use it as the foundation and skip questions already answered there. Gather the rest from stakeholders across engineering, product, and operations.

### Revenue-Critical Flows
- Which user flows directly generate revenue? (checkout, subscription, billing, upgrades)
- What is the revenue impact per hour of downtime for each flow?
- Are there time-sensitive flows? (flash sales, market-hours trading, payroll deadlines)
- Which flows have contractual SLAs with financial penalties?

### Recent Failures
- What broke in the last 3 releases? What escaped to production?
- What were the root causes? (code defect, config error, third-party failure, data migration)
- What was the blast radius of each incident? (users affected, revenue lost, reputation impact)
- Were there near-misses caught late in testing that could have escaped?

### Fragile Areas
- Which parts of the codebase change most frequently? (high churn = high risk)
- Which modules have the lowest test coverage today?
- Which areas have the most complex business logic or the most conditional branches?
- Which code was written by engineers who have since left the team?

### Third-Party Dependencies
- Which external services does the product depend on? (payment processors, auth providers, CDNs, APIs)
- What is the historical reliability of each dependency?
- What happens when each dependency goes down? (graceful degradation or hard failure?)
- Are there single points of failure with no fallback?

### Compliance and Data
- What regulatory requirements apply? (GDPR, PCI-DSS, HIPAA, SOC2, SOX, EU AI Act)
- What data is most sensitive? (PII, financial, health, credentials)
- What are the legal consequences of a data breach or compliance violation?
- Are there audit requirements that mandate specific testing evidence?

---

## Core Principles

1. **Not all features are equal.** A bug in checkout that blocks purchases is categorically different from a misaligned icon on a settings page. Equal coverage everywhere wastes resources on low-risk areas while leaving critical paths under-tested.
2. **Risk = Impact × Probability.** Risk is not a gut feeling. It is a product of two dimensions scored independently: how bad if this fails (impact), and how likely to fail (probability). Score both consistently across the product, then multiply.
3. **Risk assessment is continuous.** A risk model created once and never updated creates false confidence. Risk changes when the product changes, when dependencies change, when the team changes, and after every production incident. Build reassessment into the rhythm.
4. **Near-misses are data.** A catastrophic bug caught in staging is not a success story — it is a signal that the model underestimated that area. Track near-misses with the same rigor as production incidents.
5. **Risk informs coverage, not the other way around.** Do not start with "we need 80% coverage everywhere." Start with "where would a failure hurt most?" and let the model drive coverage targets per module.

---

## Workflow

```
1. Identify → 2. Classify → 3. Analyze → 4. Heatmap → 5. Coverage → 6. Reassess → (repeat)
                                 ↑
                       score ≥ 10 only; skip to 4 if no items reach threshold
```

### Phase 1: Risk Identification

Enumerate everything that could go wrong. Cast a wide net. Sources include:

- **Stakeholder interviews:** Product managers know business-critical flows. Engineers know fragile code. Support knows recurring user complaints.
- **Incident history:** Past failures predict future failures. Review post-mortems from the last 6–12 months.
- **Dependency mapping:** List every external service, database, message queue, and third-party API. Each is a risk vector.
- **Change analysis:** Areas with frequent code changes have higher defect probability. Use the ranked churn command in Phase 6 to find them; `git log --stat <file>` is for per-commit inspection of a specific suspect, not for ranking.
- **Architecture review:** Shared databases, single points of failure, synchronous chains, and tightly coupled modules amplify blast radius.

Use **HTSM v6.3** (Heuristic Test Strategy Model, Bach) as a Phase-1 lens — its state-based and boundary heuristics surface risks a pure feature-list misses. Download: https://www.satisfice.com/download/heuristic-test-strategy-model

Output: a raw list of risk items, each describing what could fail and what the consequence would be.

### Phase 2: Risk Classification

Categorize each risk item along two axes.

**Impact categories (how bad is it):**

| Score | Level | Definition | Examples |
|-------|-------|-----------|----------|
| 5 | Catastrophic | Revenue loss, data breach, legal action, user safety | Payment processing fails, PII exposed |
| 4 | Major | Significant user impact, SLA violation, major feature broken | Login broken for segment, data corruption |
| 3 | Moderate | Workflow disrupted, workaround exists | Search returns wrong results, export fails |
| 2 | Minor | Cosmetic or minor UX issue | Alignment bug, slow non-critical page |
| 1 | Negligible | No user impact, internal only | Admin tooltip wrong, log format issue |

**Probability categories (how likely is it):**

| Score | Level | Definition | Indicators |
|-------|-------|-----------|-----------|
| 5 | Frequent | Expected in most releases | High code churn, no tests, complex logic |
| 4 | Likely | Will probably happen within a quarter | Recent changes, partial coverage, known tech debt |
| 3 | Possible | Could happen, has happened before | Moderate complexity, some coverage |
| 2 | Unlikely | Improbable but not impossible | Stable code, good coverage, simple logic |
| 1 | Rare | Requires exceptional circumstances | Well-tested, rarely changed, simple |

Composite score = Impact × Probability. Frequent changes indicate defect probability, so a Moderate-impact (3) feature under heavy churn scores Probability 5 → **Risk score: 15 → CRITICAL zone**, despite "only" moderate impact. The composite score drives priority, not impact alone.

### Phase 3: Failure Mode Analysis

For each high-risk item (score ≥ 10), perform a detailed failure mode analysis.

```
Feature/Component: [name]
Risk Score: [impact × probability]

Failure Mode 1: [what specifically can fail]

From ExelAlfanso/puppets, MIT.

More from ExelAlfanso/puppets