evidence-driven-testing

Files (2)

• SKILL.md
• evidence.py · scripts/

SKILL.md (first lines)

# Evidence-Driven Testing

Record annotated proof of behavior, then attach it to the PR and tracker issue.

The recording is the capture of you testing the app via computer use: start the
recorder, then drive the app yourself — click, type, navigate — through each
test target. Every action in the video is the test being performed live; the
recording has no value as evidence unless it shows that interactive session.
If the harness has no computer-use tools but a GUI exists, drive the app with
`cua-driver` instead (see below) — it is still your live session.

The bundled recorder, `scripts/evidence.py`, captures the display with FFmpeg
on Linux, macOS, and Windows, timestamps each annotation you add while
testing, burns them into the video on stop, and verifies the result with
ffprobe. It writes `evidence.mp4`, `report.md`, and `manifest.json` into the
session folder.

## Inputs

- **Test targets** (required): The behaviors/flows to verify, phrased as testable statements.
- **PR / issue** (optional): Where to post the evidence. If omitted, deliver to the requester only.

## The recorder

`EVIDENCE` below means the path to `scripts/evidence.py` inside this skill's
folder (wherever the skill is installed, e.g.
`~/.claude/skills/evidence-driven-testing/scripts/evidence.py`). It needs only
Python 3 and FFmpeg.

- **Check first**: `python3 $EVIDENCE doctor` — verifies `ffmpeg`, `ffprobe`,
  `libx264`, and the `ass` filter (`ready`), then which screen-capture source
  works on this machine (`capture_ready` and the source `auto` will pick).
  Exits non-zero only when the toolchain is missing; read `capture_ready`
  before recording.
- **Platforms** (`--source auto` picks the first available):

  | OS | Source | Needs |
  |---|---|---|
  | Linux X11 / XWayland | `x11` (x11grab) | `DISPLAY` set |
  | Linux Wayland | `wayland` (wf-recorder) | `WAYLAND_DISPLAY` set, `wf-recorder` on PATH, and a compositor confirmed to support wlr-screencopy — either a known wlroots one (Sway, Hyprland, river, Wayfire, labwc, dwl, niri) or verified via `wayland-info`. GNOME and KDE Wayland are not capturable this way; `doctor` says so. Use `x11` through XWayland for X11 apps, or a fallback recorder. `--source wayland` still forces it |
  | macOS | `avfoundation` | Screen Recording permission granted to the terminal / agent host app; `doctor` lists screen indexes for `--screen-index` |
  | Windows | `gdigrab` | any standard ffmpeg build; `powershell` for process checks |

  Capture is the full screen by default; `--geometry WxH` and `--offset X,Y`
  crop a region on x11, wayland, and gdigrab. XWayland only sees X11 windows,
  so prefer the `wayland` source when the app under test is Wayland-native.
- **Crash-safe raw capture**: the raw recording is MPEG-TS (`raw.ts`), so if
  the recorder is killed hard or crashes, what was captured still probes and
  renders. `stop` remuxes or re-encodes it into a standard `evidence.mp4`.
- **How stopping works**: on Linux the recorder is signalled through a pidfd.
  On macOS and Windows `start` launches a small supervisor process that owns
  the ffmpeg child, and `stop` asks it (via `stop.request` in the session
  folder) to interrupt, then terminate, then kill; it writes
  `recorder-exit.json` when done. Nothing ever signals a bare PID, so a
  recycled PID can't be hit. If the supervisor dies while the recorder is
  still running, `stop` refuses and tells you which PID to stop by hand. If
  it died before it could even record which process it started, `stop` stays
  blocked until you have checked for a stray recorder yourself and rerun it
  with `--accept-untracked-recorder`; the report then carries that caveat.
- **Fallback recorders** when `doctor` reports no capture source (for
  example Wayland without wf-recorder): `cua-driver recording start <dir>` /
  `stop` (see the cua-driver section), or the OS recorder (macOS:
  `screencapture -v out.mov`). On these paths there is no annotation overlay,
  so keep the annotation protocol as files — an `assertions.md` listing each
  `setup` / `test_start` / `assertion` with its result and the approximate
  video timestamp, exactly as in the headless path.
- **Never** present `--source test` (the synthetic pattern generator) as UI
  evidence. It exists to smoke-test the toolchain; the repo's
  `tests/test_evidence.py` exercises it.

## Instructions

### 1. Prepare the screen

- Maximize the browser/app window; close popups, notifications, and extra panels.
- Navigate to the starting state (logged in, correct page) BEFORE recording, unless setup itself is under test.
- Note the exact revision under test: `git rev-parse HEAD` and
  `git branch --show-current` (or the deployment URL) — the recorder stamps
  them into the report.

### 2. Start recording

- Begin the screen recording before the first meaningful action:

  ```bash
  python3 $EVIDENCE start \
    --output .artifacts/<task-name> \
    --title "<what is being verified>" \
    --commit "$(git rev-parse HEAD)" --branch "$(git branch --show-current)" \
    --environment "<OS / browser / display / deployment>"
  ```

  It prints JSON with a `session` path and the chosen `source`; keep the
  path (`SESSION=...`) for every later command. The source is auto-detected
  and the whole screen is captured; pass `--source`, `--geometry`,
  `--offset`, `--display`/`--xauthority` (X11), `--screen-index` (macOS), or
  `--output-name` (Wayland) only when `doctor` or the situation calls for it.
- Add a `setup` annotation describing the starting context:

  ```bash
  python3 $EVIDENCE annotate "$SESSION" --type setup \
    --message "Logged in, navigating to connectors page"
  ```

### 3. Test via computer use, annotating as you go

- Perform every interaction through computer use on the live app — the
  recording captures your session, so the testing and the evidence are the
  same act. Work at a watchable pace: let the UI settle after each action so
  state changes are visible on video.
- At each named test's start, add a `test_start` annotation in Jest style:

  ```bash
  python3 $EVIDENCE annotate "$SESSION" --type test_start \
    --message "It should execute the tool directly when permission is 'always'"
  ```

- After each check, add an `assertion` annotation with `--result passed`,
  `failed`, or `untested`:

From kausthubh-coder/skills, MIT.

More from kausthubh-coder/skills