testing

Files (4)

• SKILL.md
• common-mistakes.md · anti-patterns/
• investigation-checklist.md · checklists/
• output-validation.md · validation/

SKILL.md (first lines)

# Nexus Flaky Test Root Cause Analyzer

Structured protocol for diagnosing, isolating, and permanently eliminating non-deterministic test
failures. Not a retry wrapper — treats flakiness as a first-class defect.

```
name:             testing
category:         testing / quality
required_context: test file path, failure frequency, CI log or local output, stack trace if available
expected_inputs:  test name, framework, failure pattern, environment (local/CI/both), reproduction rate
expected_outputs: flakiness type, reproduction steps, root cause (one sentence), narrowest fix +
                  verification command, prevention recommendation
```

## Core Principles

1. A test failing 1-in-20 runs is still failing — it is a false alarm factory.
2. A retry without investigation ships unknown risk.
3. Fix the root cause, not the symptom — masking a race condition makes it fail more often.
4. Never mark `xfail`/`skip` without a linked issue and expiry date.
5. Test flakiness often mirrors a real production concurrency or data integrity hazard.
---

## Step 1 — Collect Context

Required before any investigation:

| Signal | How |
|--------|-----|
| Failure rate | Estimate from CI history (1/5? 1/100?) |
| Stack trace | Full verbatim trace from a failing run |
| Test path | Exact: `tests/users/test_create.py::test_create_user` |
| Framework + version | `pytest --version`, `jest --version`, `go version` |
| CI vs local | Fails only in CI, only locally, or both? |
| Parallelism config | `-n auto`, `--workers`, `t.Parallel()`? |
| Recent changes | `git log --oneline -10` on test file and its imports |

**If CI-only failure**, diff these before reading code: runtime version, OS/arch, parallelism,
`TZ` / NTP, network access, env vars (missing vars silently produce defaults), I/O speed, Docker layer cache.

---

## Step 2 — Classify Flakiness Type

| Type | Signature | Mechanism |
|------|-----------|-----------|
| Ordering | Fails after specific other tests; passes alone | Shared mutable state not reset |
| Timing | Fails on slow machines or under load | Hard-coded delays, no backoff, wall-clock assertions |
| Resource | Fails parallel, passes single-threaded | Port conflicts, shared temp dirs, DB row locks |
| Environment | Fails in CI only | Runtime delta, missing env var, OS behavior |
| External | Network errors, connection refused, timeouts | Real HTTP/DB calls in test |
| Concurrency | Assertion error varies each run; stack trace differs | Thread/async race, missing lock |

**Heuristic:**
- Fails when run alone → Timing / Resource / External / Concurrency
- Passes when run alone → Ordering (shared state from another test)
- Fails only in CI → Environment delta
- Failure message varies → Concurrency or External
- Fails more under parallel load → Resource contention or Concurrency

Record: `Primary: <type> | Secondary: <type or none>`

---

## Step 3 — Reproduce

Run at minimum 5 times before investigating. Confirm flaky, not consistently failing.

```bash
pytest tests/path/to/test.py::test_fn -xvs                          # alone, verbose
pytest tests/ --randomly-seed=last -x                                # ordering check
pytest tests/ -n auto -x                                             # contention check
for i in {1..20}; do pytest tests/path/to/test.py::test_fn -x --tb=no -q; done | grep -c FAILED
```

```bash
go test ./pkg/... -run TestFn -count=20 -v 2>&1 | grep -E "PASS|FAIL"  # Go
jest --testNamePattern "name" --runInBand --verbose                       # Jest
```

Record: X/N failed, fails alone or suite-only, rate under parallelism, verbatim stack trace.

---

## Step 4 — Identify Root Cause

**Ordering:** find unreset shared state — `grep -n "global\|@classmethod\|setUp\|tearDown\|autouse"`;
look for DB rows not rolled back, in-memory caches, module-level singletons, `scope="session"` fixtures.

**Timing:** `grep -rn "time.sleep\|asyncio.sleep\|setTimeout"` and polling loops without timeout;
look for fixed delays insufficient on slow machines, wall-clock assertions.

**Resource:** `grep -rn "port.*=[0-9]\{4,5\}\|localhost:[0-9]\{4,5\}"` and `grep -rn "tmp\|tempfile"`;
look for hardcoded ports, shared temp dirs, DB connections without rollback.

**Environment:** `cat .github/workflows/*.yml | grep -E "python-version|node-version|go-version"`;
compare env vars between local and CI.

**External:** `grep -rn "requests\.\|httpx\.\|fetch(\|http.Get"` and real DB connection strings in tests.

**Concurrency:** `grep -rn "threading\.\|asyncio\.\|goroutine"` and missing `Lock`/`Mutex`/`await`.

**State root cause as exactly one sentence:**
> *"test_X fails intermittently because Y when Z."*

If you cannot state it in one sentence, root cause is not yet identified.

---

## Step 5 — Fix

Apply the narrowest fix. Do not improve unrelated code.

| Type | Wrong fix | Correct fix |
|------|-----------|-------------|
| Ordering | Delete the interfering test | Add teardown resetting shared state after each test |
| Timing | Increase `time.sleep(N)` | Replace with `wait_for(condition, timeout=N)` |
| Resource | Manually assign different port | Use `port=0` or a `free_port()` fixture |
| Environment | Hard-code CI env locally | Parametrize via env var; test both modes in CI |

From aayushostwal/nexus, MIT.

More from aayushostwal/nexus