scripted-verification

Files (2)

• SKILL.md
• verdict-skeleton.sh · scripts/

SKILL.md (first lines)

# Scripted Verification

Agents are expensive judges. Every step re-sends the whole context, so an agent
that re-reads results in prose pays for that context on every step. One audit
found the largest cost was agents re-judging the same kind of result: about 29
runs in a day, some for 25 to 40 minutes at high effort, at 110K to 134K tokens
per step. A script makes that judgement once, for free, every time.

## When To Reach For This

Stop and script the check when you notice yourself:

- about to read logs, screenshots, diffs, or numbers to decide pass or fail,
  especially the same comparison against the same baseline as last time;
- writing a "check X", "verify X", "retest", or "make it look right" brief
  for another agent;
- following a manual check from AGENTS.md, a PR template, or a review comment;
- about to sleep, wait, or poll until a run, build, or resource finishes;
- re-checking a commit, or a commit whose inputs did not change, that already
  has a verdict;
- launching a costly check that fails for a cheap reason: merge conflicts, a
  busy shared resource, or missing arguments;
- arguing whether a borderline number is noise. Measure the noise floor
  instead.

## Script Any Check That Runs More Than Once

The second time an agent judges the same kind of result, write a script.

- Print one verdict line: `PASS`, `FAIL`, or `UNSURE`, with the evidence
  path. Use distinct exit codes: `0` pass, `1` fail, `2` unsure, `3` the
  check itself broke.
- Write the full evidence (screenshots, diffs, raw numbers, logs) to `--out` or
  a debug directory. Do not print it.
- Use `UNSURE` for anything that is not a clean pass or fail: the borderline
  band and states the script cannot decide. Do not force a guess.
- Agents read the verdict line. On `UNSURE`, an agent reviews the saved
  evidence first. Involve a person only when that review still cannot decide.
- Every verdict stays reviewable: the evidence must show why the script decided
  what it did.

Skeleton: [`scripts/verdict-skeleton.sh`](scripts/verdict-skeleton.sh).

## Set Thresholds From Measured Noise

- Measure the noise floor first: run the check several times on an unchanged
  state and record the spread. Set the threshold above it, with an
  `UNSURE` band between noise and a clear failure.
- Do not lock references or goldens until the state is stable. A golden taken
  during churn turns every later run into noise.

## Pre-Check Before Launching An Agent

Run cheap checks first and skip the agent when one fails:

- the branch has merge conflicts;
- a shared resource (GPU, machine, lock) is busy;
- the commit was already judged. Cache verdicts by commit and inputs, and
  reuse the cached verdict.

## Watch Events Instead Of Polling Agents

Wait with CI, webhooks, file watchers, or a cheap script loop. Never keep an
agent idling in a wait loop; wake one only when there is a result to act on.

## Earn Trust

- A new check starts on probation: review its verdicts against the evidence.
- It graduates to trusted after a run of correct verdicts.
- Any miss, false pass, or false fail becomes a new test case for the check.

## Brief Agents Tightly

- Give short, focused tasks. Start a fresh agent per task instead of growing
  one long context.
- Use targeted reads: the failing section or the verdict, not whole files or
  logs.
- Write a precise brief: the goal, the inputs, the done criteria, and the
  exact command to run.

## Related

- `context-management` for targeted reads and handoffs.
- `create-verification-skill` for driving an app and collecting evidence.
- `lint-rules` and `encode-lessons` for turning a repeated judgement into a
  guard.
- `test-audit` for whether a check should exist at all.

From AnthonyPoschen/agent-skills, MIT.

More from AnthonyPoschen/agent-skills