invariant-test

Files (1)

• SKILL.md

SKILL.md (first lines)

# Invariant Test

Turn a service's **stated invariants** into surface tests that **fail when the invariant is violated** — the tests that integrity-only suites and the race detector cannot produce.

The motivating case: querator's single-writer-per-partition invariant (ADR-0003, restated load-bearing at `internal/logical.go:97-100`) was violated by the DLQ-movement path, yet `-race` was green and every test passed. It survived because **no test could fail when single-writer broke** — every DLQ test asserted *integrity*, which the defensive per-backend mutexes provide for free. This skill manufactures the assertions those mutexes cannot satisfy (liveness, timeliness, ordering, ownership) and proves each one catches the violation before declaring it done.

**Three principles run through the whole pipeline:**

1. **`-race` green proves memory safety, not architectural correctness.** A data race and an invariant violation look identical at the diff — both are "two goroutines touch this resource" — but they have opposite fixes (add a lock vs. remove the second writer). A clean race run and a tidy mutex actively *hide* the architectural violation. This skill never treats a passing race/integrity test as evidence an invariant holds.

2. **A correctness test you have not watched go red is not yet a correctness test.** Every test this skill writes is **mutation-validated**: deliberately break the invariant, confirm the test turns red, restore, confirm it turns green. A test that stays green when the invariant is violated is rewritten until it doesn't — or the invariant is flagged as needing new observability. This is non-negotiable and is the load-bearing phase.

3. **Test the invariant's *purpose*, never its mechanism.** You cannot assert "only one goroutine wrote this partition" at the surface, and `-race` can't either. You assert the behavior the invariant *exists to guarantee* (the waiter is woken, the timer advances now, ordering holds) — observed through the public surface per the `surface-testing` skill.

This skill requires a filesystem, the source code, and the ability to run the test suite. It is Claude Code only. It writes the proving test; it does **not** fix the implementation.

## How this differs from neighboring skills

- **`review-tests`** measures coverage of *spec acceptance criteria* (PRD / tech spec / user stories) and fills gaps. `invariant-test` starts from *invariants* (ADRs + load-bearing doc comments), derives the consequences of violating them, and writes the one assertion a defensive mechanism can't satisfy. Where `review-tests` asks "is this criterion tested?", this asks "would any test fail if this invariant broke?" Running `review-tests` first and `invariant-test` second is a reasonable pairing.
- **`review-validate`** validates a list of *already-identified bugs*, proves each with a failing test, then **fixes** them. `invariant-test` discovers the *missing test class* from the design contract, and when a test is red-now it reports a **live invariant violation** loud but does not fix it — hand the fix to `review-validate`.
- **`review-behavior-change`** flags *undocumented* changes to public behavior in a diff. `invariant-test` works from the *documented* invariants toward tests, regardless of any diff.
- **`surface-testing`** is the philosophy every test here obeys. Read it; this skill assumes its rules (public interface only; expose observability when async behavior isn't visible at the surface).

## Inputs

`$ARGUMENTS` may be any of:

- **A feature directory** (e.g. `docs/features/<name>/`) — scope invariants to that feature's tech spec plus the ADRs and doc comments its implementation touches.
- **A specific invariant or issue** stated in the prompt or referenced (a Linear/GitHub issue like ENG-55, an ADR number, a quoted doc comment). Build the ledger from that one invariant and go deep.
- **Nothing** — scan the repository: every Accepted ADR plus load-bearing doc comments across the codebase. Confirm scope with the user before fanning out, since a whole-repo scan can be large.

Before starting, locate and read what exists:

- **`docs/adr/`** — list every ADR; read the **Accepted** ones (skip Proposed/Superseded/Rejected unless the user asks). ADRs are the primary invariant source.
- **`docs/CONTEXT.md`** (walk up from any feature dir) — domain glossary; use it to map invariant terms onto code, surface, and stats names.
- **Tech specs** in the target feature directory, if scoped to a feature.
- **`invariant-test.html`** — the report this skill wrote on prior runs, if present at the repo or feature `docs/` root. Parse its embedded JSON (`<script type="application/json" id="invariant-test-data">`) to recover the `runs` log and each invariant's status history. Use it to (a) avoid re-deriving invariants already covered by a `mutation-verified` test, (b) detect regressions where a previously-proven invariant test was deleted or now passes-under-mutation (stopped catching the violation), (c) carry forward open `needs-observability` items, and (d) **re-examine, never trust, any prior verdict whose `evidenceGrade` is not `mutation-verified`.** A reasoning-only "no surface consequence" verdict that calcifies will suppress re-investigation forever; only a test that was written and watched go red-then-green is carried forward without re-probing.

If no ADRs, doc-comment invariants, or tech specs can be found, tell the user: there is no *stated* correctness to test against. Offer to elicit the invariant directly ("what must always/never be true here?") or to record it as an ADR first (`adr-write`) — an unstated invariant cannot be checked and will simply be re-broken.

## Phase 0: Scope, Surface & Baseline

1. **Locate invariant sources.** Read the Accepted ADRs in scope. Then grep the codebase for **load-bearing doc comments** — the contract vocabulary that signals a stated invariant:
   `MUST`, `ONLY`, `single`, `exactly one`, `never`, `always`, `invariant`, `synchronization point`, `singleton`, `single-threaded`, `at most one`, `owns`, `must go through`.
   Also capture the **bypass/hedge vocabulary**, which marks where an invariant is *deviated from* (these are both invariant evidence and prime suspects for live violations):
   `directly`, `bypass`, `without going through`, `defensively`, `fast path`, `internal`, `should only`, `normally`.
2. **Identify the defensive mechanisms.** For each resource an invariant governs, find what already guards it: mutexes, RWMutex, atomics, DB transactions, individually-atomic backend ops, channels. **Record these per resource** — they decide which assertion buckets are masked in Phase 3. This is unique to this skill and you cannot skip it.
3. **Identify the surface.** Per `surface-testing`, list the public entry points (HTTP/gRPC handlers, CLI `Run()`, exported client methods, stats/observability endpoints). Every test enters here. Note the observability surface specifically (e.g. a `Stats()` / `QueueStats` API) — invariants whose consequences are otherwise invisible are tested through it.
4. **Run the test suite once** for a baseline, exactly as CI does (e.g. Go: `make ci` or `go test ./... -race -p=1`). Capture pass/fail. A green baseline is the backdrop against which a red-now invariant test stands out as a live violation.
5. **State scope to the user** in a few lines: which invariants are in scope, which resources/defensive mechanisms were found, which surface, and whether the baseline is green.

## Phase 1: Invariant Ledger

Build a numbered ledger. Do it yourself, or delegate to one extraction sub-agent if the ADR set is large. Each entry:

- **id** — `INV-1`, `INV-2`, … Carry forward ids from a prior `invariant-test.html` when the invariant is unchanged so history lines up.
- **statement** — the invariant in one sentence ("each storage partition has exactly one writer: its Logical's `requestLoop` goroutine").
- **source** — exact citation (ADR number + section, or `file:line` of the doc comment). Quote it.
- **resources** — the shared state/resource(s) it governs (the DLQ partition; `p.State`).
- **defensive mechanisms** — what currently guards each resource (from Phase 0 step 2), e.g. "`MemoryPartition.m.mu` on every method; Mongo ops individually atomic."
- **purpose** — *why the invariant exists / what it protects.* This is what you will actually test. ("The single writer is the only thing that, in one pass, keeps `p.State` consistent with storage **and** wakes blocked waiters **and** advances the lifecycle timer.")

Skip nothing that is stated. If a doc comment and an ADR state the same invariant, record one entry citing both.

## Phase 2: Consequence Derivation (parallel sub-agents)

Partition the ledger into balanced batches and spawn a small number of sub-agents in parallel (Agent tool, **foreground**, `model: "sonnet"`; default general-purpose agent for full tool access). For each invariant, derive **what a user would observe if it were violated**, bucketed:

- **integrity** — data wrong, lost, duplicated, corrupted.
- **liveness** — something that should happen never does / hangs (a blocked waiter never woken).
- **timeliness** — happens late / drifts (a timer not advanced; state stale until an unrelated wake).
- **ordering** — sequence/causality broken.
- **other** — uniqueness, idempotency, monotonicity, etc.

Sub-agent prompt structure:

```
You are deriving the observable consequences of violating a stated invariant.
For each invariant below: read the cited source and the code around the governed
resource yourself. Then answer, concretely and in terms of the PUBLIC SURFACE only:

For each consequence bucket (integrity / liveness / timeliness / ordering / other):
- Would violating this invariant produce an observable defect in this bucket? Describe it.
- Is that defect observable through the public surface (endpoint / exported call / stats)?
  If yes: name the exact surface call and the assertion that would detect it.
  If no: say so — this becomes a candidate "needs-observability" finding.

Invariant: <id> — <statement>
Source: <citation, quoted>
Governed resource(s): <...>
Defensive mechanisms on those resources: <...>
Purpose (what it protects): <...>
Public surface available: <entry points + stats API>

Output per invariant, per bucket: { bucket, defect, surfaceObservable: bool,
  surfaceCall, proposedAssertion }.
Do NOT propose a test yet. Just enumerate consequences and how each would be seen.
```

Collect all consequences. Read the relevant source yourself before trusting an agent's "not surface-observable" claim.

## Phase 3: Mask Filter (the load-bearing critic)

**This is the step every integrity-only suite skips, and the reason ENG-55 survived.** Spawn one sub-agent (Agent tool, **foreground**, `model: "sonnet"`) — or do it directly for a small ledger — to challenge each consequence against the defensive mechanisms recorded in Phase 1.

For every consequence, ask the discriminating question:

> **Would this assertion still PASS if the invariant were violated, because a defensive mechanism (mutex / transaction / atomic op) satisfies it anyway?**

- **Yes → DISCARD.** The assertion tests the lock, not the invariant. Integrity assertions almost always fall here when a mutex or atomic backend op guards the resource ("no corruption / no lost item" passes under the broken DLQ path because the mutex serializes writes). Record the discard with its reason in the report — it documents *why* the obvious test would have proven nothing.
- **No → KEEP.** The defensive mechanism cannot make this pass while the invariant is broken. These are the surviving assertions — usually **liveness** ("the blocked waiter is woken promptly") and **timeliness** ("the lifecycle timer advanced now, not on the next unrelated wake"). They are what the test must assert.

Sub-agent prompt structure:

```
For each (invariant, consequence) below, decide KEEP or DISCARD.
DISCARD iff a listed defensive mechanism would make the proposed assertion PASS
even while the invariant is violated (i.e. the assertion tests the mechanism, not
the invariant). Otherwise KEEP. For each, give a one-line reason and, for KEEPs,

From thrawn01/claude-dotfiles, MIT.

More from thrawn01/claude-dotfiles