go-test-strength

Files (3)

• SKILL.md
• openai.yaml · agents/
• mutate.sh · references/

SKILL.md (first lines)

# Go Test Strength Skill

Read and follow **agent-conduct** and **testing-principles** before starting.

A green suite proves the implementation passes its own tests. It says nothing
about which defects those tests would notice. This skill measures that
directly: break the implementation on purpose, one named defect at a time, and
require the suite to fail.

Coverage does not answer the question. A line executes under a test that
asserts nothing about it, so a suite at 98% statement coverage can miss whole
classes of defect at a type's boundaries. Treat coverage as a place to aim,
never as evidence.

## Core Rule

Every defect you can name and inject must make the suite fail. A defect the
suite tolerates is a finding, reported with the input that exposes it.

## Verdicts

Use these three words for every mutant, and no synonyms:

- **KILLED** - the suite failed. The tests defend that behaviour.
- **SURVIVED** - the suite passed. A finding or an equivalent mutant, decided
  in step 5.
- **INVALID** - the patch did not apply, or the mutant does not compile. Not a
  result.

An INVALID mutant counted as SURVIVED is what makes this exercise lie, and the
two look identical in the test output. Step 4 separates them mechanically
because judgement will not.

## Procedure

### 1. Baseline

```bash
go test ./<pkg>/ -count=1 -coverprofile=cover.out
go tool cover -func=cover.out
golangci-lint run ./<pkg>/...
```

Done when the pristine suite passes and the linter is clean. A red or flaky
baseline makes every later verdict noise: stop and report that instead.

Record every function below 100%. Each is a place to aim in step 3.

### 2. Isolate the workspace

Copy the module skeleton and the package into the scratch directory
**agent-conduct** allows, keeping the module path so imports still resolve.
Mutate the copy, never the working tree.

```bash
W="$SCRATCH/mutants"
mkdir -p "$W/<pkg>"
cp go.mod go.sum "$W"/
cp <pkg>/*.go "$W/<pkg>/"
```

Done when `go test ./<pkg>/ -count=1` passes inside the copy too.

### 3. Write the mutant list

Three sources, in this order:

1. **The spec.** Every clause of the form "so a naive X cannot pass",
   "asserted absent", "rather than", or "asserted side by side" names a mutant
   outright. Write those first: a survivor there is an unmet requirement, not
   a suggestion.
2. **The catalogue below**, applied to each exported function.
3. **The partially covered functions** from step 1.

Done when every exported function has at least one mutant, and every "cannot
pass" clause in the spec has the mutant it names.

### 4. Run each mutant

[mutate.sh](references/mutate.sh) applies, builds, tests and restores as one
step, and prints one of the three verdicts:

```bash
references/mutate.sh "$W" <pkg> <file> old.txt new.txt "<label>"
```

Replace text literally, which is what the script does. A `sed` or `perl`
pattern silently mangles Go punctuation, and a mutant that never applied reads
exactly like a mutant the suite failed to catch.

One mutant at a time: two at once can mask each other.

Done when no mutant is left INVALID.

### 5. Triage every SURVIVED mutant

Decide mechanically whether it is a defect or an equivalent mutant. Write a
throwaway test that calls both versions across the input domain, including
type boundaries, zero, and every sign quadrant:

- One input where they differ: a defect. Report that input.
- No differing input, plus an argument for why: equivalent. Exclude it.

Read the code and judge instead, and you will file real bugs as no-ops. An
off-by-one that looks like a harmless rewrite (`x < lo` to `x < lo+1`) is a
live signed-overflow bug at the type's maximum, which is exactly where a weak
suite has no assertion.

Done when every survivor carries a written verdict of defect or equivalent.

### 6. Report

One finding per surviving defect:

- the mutant, as the exact before and after text
- the input that distinguishes it from the original
- the assertion the suite is missing, named as a test the author can write

Close with the kill rate over valid mutants. Report it as evidence, not a

From wtsi-hgi/agentskills, MIT.

More from wtsi-hgi/agentskills