testing-skills

Files (5)

• SKILL.md
• TESTS.md
• cso.md · references/
• pressure-scenarios.md · references/
• rationalization-table.md · references/

SKILL.md (first lines)

# Testing Skills

Behavioral proof for **discipline skills** — those that enforce a rule an agent
could rationalize past. Structure is not compliance. This proves the skill holds
under pressure, deterministically.

## When to Use

Use for a **discipline skill**: one whose value is that it runs even when the
agent is tempted to skip it (mandatory reviews, quality gates, process steps).
Signs: it says MUST/ALWAYS, lists "blocked rationalizations", or gates a phase.

Do **not** use for technique or reference skills — ones that inform but enforce
nothing. Those are covered by the structural scorecard; they have no skip to
prove.

## The Iron Law

**No discipline skill ships without a failing behavioral test first.** Applies to
new skills *and* edits. When you catch a discipline skill about to ship or change
without a RED→GREEN run, announce **`⚡ RED baseline first`** and run it. If you
cannot point at that run, you are not done.

## The Cycle: RED → GREEN → REFACTOR

- **RED** — run the scenario with the skill **withheld**. Record the agent's exact
  skip rationalizations, verbatim. The marker MUST be absent. If you never
  watched it fail, you don't know it prevents the failure.
- **GREEN** — run with the skill present. The marker MUST appear. That marker,
  grepped from the transcript, is the entire verdict. No agent judges compliance.
- **REFACTOR** — stack combined pressure (time + sunk-cost + exhaustion). Each new
  rationalization that drops the marker gets a counter; re-run until the marker
  holds at maximum pressure. See `references/pressure-scenarios.md`.

## The Four Requirements

Every discipline skill MUST carry all four (enforced by
`validate-discipline-skill.sh`):

1. **`compliance_marker`** — an exact string the *real* code path emits when
   followed. Not a test-only echo.
2. **`TESTS.md`** beside SKILL.md, latest `green_verdict: PASS`.
3. **`## Rationalization table`** — observed excuses + counters
   (`references/rationalization-table.md`).
4. **CSO-clean `description`** — triggering conditions only, no workflow summary
   (`references/cso.md`).

## Running the Test

`develop/skill-tests/run-skill-test.sh <skill-dir> --both` dispatches the
scenario with the skill withheld (RED) then present (GREEN). RED passes when the
marker is absent; GREEN when present. Off-platform it exits 2 with a manual-mode
instruction — a dev tool, never a runtime dependency.

## Failure Modes

- **Tempted to have an agent judge "did it comply?"** — stop. The marker grep is
  the verdict; judgment reintroduces the non-determinism this removes.
- **Marker real execution doesn't emit** — invalid; the test passes while
  production skips. Confirm it against the genuine code path.
- **Imagined rationalization table** — derive rows from real RED transcripts.

From xoai/sage, MIT.

More from xoai/sage