coverage-analysis

Files (4)

• SKILL.md
• ci-gating.md · references/
• mutation-testing.md · references/
• tool-config.md · references/

SKILL.md (first lines)

<objective>
A suite at 90% line coverage with assertion-free tests catches zero bugs — line coverage
proves code ran, not that a regression would be caught. This skill measures the right
things (branch coverage, mutation score, critical-path coverage), gates them in CI with a
ratchet so coverage can only go up, and surfaces gaps by risk instead of chasing a vanity
number. It prevents the classic failure: a green coverage badge over a test suite that
never asserts anything meaningful, while the payment module sits at 30%.
</objective>

## Quick Route

| Situation | Go to |
|-----------|-------|
| Pick and configure a coverage provider | Coverage Tools → `references/tool-config.md` |
| Decide where to write tests next | Gap Analysis |
| Stop coverage from regressing in CI | Coverage as CI Gate → Ratchet Pattern |
| Show per-PR coverage to reviewers | Coverage as CI Gate → PR Diff |
| Tests run code but don't assert | Mutation Testing |
| Decide what to exclude / what target to set | Meaningful vs Vanity Coverage |

---

## Discovery Questions

Check `.agents/qa-project-context.md` first — if it exists, use it and skip anything already answered there. Then:

1. **What test runner and coverage tooling is configured?** Check for `vitest.config.*` (coverage block), `jest.config.*` (coverageProvider), `.nycrc`, `c8` in scripts, or `[tool.coverage]` in `pyproject.toml`. The runner decides the install — Vitest pulls `@vitest/coverage-v8`, not c8 (see Coverage Tools).
2. **What is the current coverage level?** Run the existing coverage command and note line, branch, and function percentages. This is the baseline for the ratchet.
3. **Is coverage gated in CI?** Check GitHub Actions / GitLab CI for `--coverage`, `coverageThreshold`, `fail_under`, or `--cov-fail-under`. No gate means coverage is decorative.
4. **What is the target, and who set it?** A target without rationale ("the VP said 80%") leads to gaming. Targets should reflect risk tolerance and codebase maturity, not a round number.

---

## Core Principles

**1. Coverage measures breadth, not depth.** A line being executed does not mean it is tested correctly. `expect(true).toBe(true)` executes the function but asserts nothing. Coverage tells you what code ran, not whether the tests would catch a bug — that is what mutation testing measures.

**2. Branch coverage matters more than line coverage.** A ternary `condition ? a : b` on one line counts as fully covered in line coverage even if only one branch ran. Line coverage does not guarantee branch coverage. Gate on branches, not just lines:

```typescript
function discount(price: number, isPremium: boolean): number {
  return isPremium ? price * 0.8 : price;
}

// Line coverage: 100% (the line executed). Branch coverage: 50% (only the true branch ran).
expect(discount(100, true)).toBe(80);

// Fix — assert BOTH paths so branch coverage reaches 100%:
expect(discount(100, true)).toBe(80);
expect(discount(100, false)).toBe(100);
```

**3. Ratchet pattern: never decrease, only increase.** Record current coverage as the minimum threshold. Every PR must meet or exceed it. Coverage climbs over time without forcing an artificial target up front.

**4. Focus on gaps by risk, not on the number.** A project at 85% is not automatically "better" than one at 75%. What matters is whether the untested slice contains payment, auth, or data-integrity logic. Analyze gaps by risk.

**5. New code has a higher bar than legacy code.** Require 90%+ on new code in PRs even if the project sits at 65%. This stops coverage decay without demanding a rewrite of legacy code.

---

## Coverage Tools

Pick the provider by **test runner first**, then by how cleanly it maps to your build output.

| Runner / context | Install | Provider |
|---|---|---|
| **Vitest** | `@vitest/coverage-v8` (default) or `@vitest/coverage-istanbul` | `coverage.provider: 'v8'` / `'istanbul'` |
| **Jest** | bundled (`coverageProvider: 'v8'` or `'babel'`) | V8 or Istanbul/babel |
| **Non-Vitest Node** (`node:test`, plain mocha) | `c8` CLI | V8 via `c8 <command>` |
| **Legacy Istanbul CLI** | `nyc` | Istanbul instrumentation |
| **Python** | `pytest-cov` (wraps coverage.py) | coverage.py |

Two engines underneath:

- **V8 coverage** — built into the V8 engine, so it does not instrument source: faster, no Babel transform. For Vitest, install `@vitest/coverage-v8` (NOT `c8` — that is the standalone CLI for non-Vitest runners). For a plain `node:test` or mocha project, `c8` is the CLI wrapper around the same V8 data. Default for new Node/Vitest projects.
- **Istanbul** — instruments source code; slower but maps more reliably through transpilers and bundlers. Switch to it (`@vitest/coverage-istanbul`, or `nyc`) when V8 maps poorly. **Symptom that V8 maps poorly:** reported uncovered lines land on blank lines, closing braces, or decorators, or whole covered functions show as red — that means the source map is misattributing lines (common with certain TS bundlers / SWC configs). When you see that, flip to the Istanbul provider.

> **Node baseline:** `c8` 11.x and `nyc` 18.x are current. c8 11 still supports Node >=12; nyc 18 requires **Node 20 || >= 22**. If you must stay on Node 18, pin `nyc@^17` (c8 11 runs fine on Node 18). New projects should standardize on Node 20+.

See `references/tool-config.md` for the full provider configs (Vitest `coverage` block, `.nycrc.json`, Jest `coverageThreshold`, `pyproject.toml` / `.coveragerc.toml`), install commands, and run invocations.

### Merging coverage across test types

Unit, integration, and E2E runs each produce partial coverage. Combine them so a line covered only by an integration test isn't reported as a gap:

- **Vitest** — run as multiple projects/configs and let Vitest merge, or merge `coverage-final.json` outputs.
- **nyc** — `nyc merge .nyc_output merged.json && nyc report -t merged` combines `.json` files from separate runs.
- **coverage.py** — `coverage combine` after running each suite with `coverage run -p`.

Merge first, gate on the merged total. Don't gate each suite's coverage in isolation.

### Coverage Report Types

| Reporter | Output | Use Case |
|----------|--------|----------|
| `text` | Terminal table | Quick local check |
| `html` | Interactive HTML | Detailed local analysis, clicking through files |
| `lcov` | `lcov.info` file | SonarQube, Codecov, Coveralls integration |
| `json-summary` | `coverage-summary.json` | CI scripts, PR comments, dashboard metrics |
| `cobertura` | `cobertura-coverage.xml` | GitLab CI coverage visualization |

---

## Gap Analysis

Coverage reports show which lines and branches did not run. Not all gaps are equal — prioritize by risk.

**Step 1: Generate the report.**

```bash
npm run test:coverage
# Open coverage/index.html in a browser
```

**Step 2: Sort files by uncovered lines.** Parse `coverage-summary.json`, sort files by `(total - covered)` descending, focus on the top 20. A small script that reads the JSON summary and outputs file / line% / branch% / uncovered-count makes this repeatable.

**Step 3: Map gaps to risk.**

| Gap Location | Risk Level | Action |

From ExelAlfanso/puppets, MIT.

More from ExelAlfanso/puppets