verify-with-playwright

Files (1)

• SKILL.md

SKILL.md (first lines)

# verify-with-playwright

A contributor workflow for AI Atelie. Drives the local dev server with `@playwright/test` (runner mode, not MCP), produces a `test-results/<spec>/` folder with video + screenshots + trace, then bundles the relevant files into `.evidence/<run>/` for upload to a PR.

This skill is **dev-time only**. It does not load into adapter sessions spawned by the editor.

## When to invoke

- After implementing any change that touches `web/`, the iframe canvas, the chat surface, the editor toolbar, or any other user-visible behaviour.
- When the `ship-task` orchestrator delegates verification.
- When the user asks "does it actually work in the browser?" or "show me proof".

Skip when:

- The change is API-only with no observable surface (use `web/tests/test-*.mjs` integration tests instead).
- The change is a doc-only or config-only edit.

## Preflight (do this before writing any spec)

1. **Confirm the dev server is up.** Run:
   ```sh
   curl -sSf http://localhost:5173 > /dev/null && echo "up" || echo "down"
   ```
   If down, **STOP** and tell the user: "Run `bun run dev` in another terminal, then retry." Do NOT spawn `bun run dev` from this skill — backgrounded dev servers leak across sessions and will clobber the user's own terminal.
2. **Confirm Playwright is installed.** `bunx playwright --version` should print `Version 1.59.x` or higher. If not, run `bun add -D @playwright/test && bunx playwright install chromium`.
3. **Read `playwright.config.ts`** at repo root to confirm the test dir, baseURL, and artifact policy haven't drifted.

## Workflow (paste this checklist into your reply and tick as you go)

- [ ] **Preflight passed** (dev server up, Playwright installed).
- [ ] **Acceptance criteria identified.** From the issue or the user's task description, list the *user-observable* properties that must hold. Example for #1 (live preview): "agent text deltas appear in iframe srcdoc within 500ms of arrival; full reload not required".
- [ ] **Spec authored.** Write `web/tests/e2e/<slug>.spec.ts`. One scenario per file. Use role-based selectors. Each acceptance criterion → one or more `expect(...)` calls.
- [ ] **Spec runs.** `bunx playwright test web/tests/e2e/<slug>.spec.ts --reporter=list`.
- [ ] **Wait for the agent to FULLY COMPLETE before stopping the recording.** This is the rule reviewers care about most — see "Evidence must show the post-completion state" below. A capture that ends mid-stream proves nothing.
- [ ] **Capture the post-completion state explicitly.** A final screenshot AFTER the busy/loading state clears AND the user-visible result is rendered. The reviewer must be able to see *what the agent actually produced*, not just *that it was working*.
- [ ] **Evidence bundled.** Copy `test-results/<slug>/video.webm` and any `*.png` into `.evidence/<ISO-timestamp>-<slug>/`.
- [ ] **(Optional) headed run for video review.** `--headed` for clearer screen recordings on visual changes.
- [ ] **Cleanup.** Prune `.evidence/` to keep only the last 5 run folders.
- [ ] **Report.** Output the absolute paths of evidence artifacts so `ship-task` (or the user) can attach them to the PR.

## Authoring a spec

Minimal scaffold:

```ts
import { test, expect } from "@playwright/test";

test("live-preview streams text deltas into iframe", async ({ page }) => {
  await page.goto("/");
  // ... drive the surface that exercises the change

  // ... assert acceptance criteria
  await expect(page.frameLocator("iframe").locator("h1")).toHaveText(/Hello/);
});
```

Conventions:

- **Always `await page.goto('/')` first.** The `baseURL` resolves to `http://localhost:5173`.
- **Prefer role-based selectors** (`getByRole('button', { name: 'New project' })`) over CSS selectors. They survive markup churn.
- **One scenario per file**, named `<feature>.spec.ts`.
- **Test the surface, not the agent's output.** The agent is non-deterministic; the surface's behaviour around it should be deterministic.
- **No external network in tests.** Mock the agent stream with `page.route('/api/...', route => route.fulfill({...}))` if needed.

## Running the spec

```sh
# Full run, list reporter (Claude reads this output well)
bunx playwright test web/tests/e2e/<slug>.spec.ts --reporter=list

# Headed mode for better video evidence
bunx playwright test web/tests/e2e/<slug>.spec.ts --headed

# View the HTML report after a failure
bunx playwright show-report
```

## Evidence MUST show the post-completion state — including the canvas

**Rule**: a capture that ends while the agent is still working, OR that doesn't include the visual outcome on the canvas / iframe / final UI surface, is incomplete evidence and should never be submitted to a PR.

The point of evidence is to demonstrate, in order:

1. **The change produces the expected user-visible outcome.** The agent finished, the result rendered, the **iframe canvas shows the new content** (or the file was written, or the UI updated to reflect the new state) — whatever the acceptance criteria says is "the thing the user will see."
2. (Optionally) **The chat / chip / loading UI was correct in flight** — busy pill rotated, chip turned amber while editing, etc.
3. (Optionally) **Failures are valid evidence too.** If the agent's API errored, hit a rate limit, or refused — capture that screenshot and explain. It's better than no evidence; reviewers can tell whether the feature works or whether the agent had a bad day.

If your screenshot only shows the chat sidebar without the canvas, you've captured *the harness reacting* but NOT *the user-visible result*. Reviewers can't tell whether the feature works.

### Pick prompts that produce a visible canvas change

For features whose acceptance criteria is "the chat sidebar / chips / status indicator does X," a pure-chat capture is fine. But the **default** evidence prompt should be one that ALSO updates the iframe canvas, so the same screenshot proves both surfaces work:

- ✅ `"Make this a Hello World page on a bright green background. Replace whatever is there."` — agent writes index.html, canvas re-renders green, chip strip + canvas both visible.
- ✅ `"Add a red rectangle in the top-right corner of the canvas."` — visible diff on the iframe.
- ❌ `"Just read the current index.html."` — chat updates, canvas unchanged. You proved the *chat* works, not the *product*.

If the acceptance criteria genuinely is chat-only (e.g., "tool chips are colored"), a chat-only capture is acceptable but you should still trigger a chat that *would* have updated the canvas — that way the still-blank canvas is itself evidence that the agent ran in a normal mode, not a special test mode.

### How to wait for completion correctly

Don't sleep a fixed timeout — poll for the *signal* that completion happened:

```ts
// Wait for the agent's busy state to CLEAR after we know it started.
// (Make sure we saw it busy at least once first — otherwise we'd race
// past a fast turn that finished before we started polling.)
let sawBusy = false;
const start = Date.now();
while (Date.now() - start < 6 * 60_000) {
  const busy = await page.locator("[role=status]").isVisible().catch(() => false);
  if (busy) sawBusy = true;
  else if (sawBusy) break;
  await page.waitForTimeout(2_000);
}
expect(sawBusy, "agent run was observed to be in flight at some point").toBe(true);
```

Then add ONE extra `await page.waitForTimeout(2_000)` after the busy state clears so the iframe re-renders before you screenshot.

From aiatelie/ai-atelie, MIT.

More from aiatelie/ai-atelie