vibe-testing

Files (6)

• LICENSE
• README.md
• SKILL.md
• example-vibe-test.md · examples/
• marketplace.json
• simulator-prompt.md · references/

SKILL.md (first lines)

# Vibe Testing

## Overview

Vibe testing validates specification documents by simulating real-world scenarios against them using LLM reasoning. Instead of writing code or test harnesses, write **natural-language scenarios** that exercise cross-cutting slices of the spec surface — then trace execution step-by-step, flagging gaps, conflicts, and ambiguities.

**Core principle:** If a realistic user scenario cannot be fully traced through the specs, the specs are incomplete.

**Best used:** After specs are written, before implementation begins.

## When to Use

- Spec docs exist but no implementation yet — validate before building
- After major spec changes — regression-test for new gaps
- Before implementation planning — find blocking gaps early
- Specs span multiple documents — test cross-doc coherence
- Designing for multiple deployment contexts — test each context separately

**When NOT to use:**

- Single-file specs with obvious scope — just review manually
- Implementation bugs — use actual tests
- API contract validation — use schema validation tools

## Core Method

```
1. GATHER    — Read all spec docs in the target directory
2. SCENARIOS — Write 3-5 vibe test cases (personas + goals + environments)
3. SIMULATE  — Trace each scenario step-by-step against the specs
4. CLASSIFY  — Tag findings as GAP / CONFLICT / AMBIGUITY
5. SEVERITY  — Rate as BLOCKING / DEGRADED / COSMETIC
6. REPORT    — Produce gap summary + spec coverage matrix
```

## Writing a Vibe Test Case

Every test case requires 7 sections:

### 1. Persona (WHO)

A concrete person with a name, role, and technical skill level. Not abstract — real enough to predict behavior.

```markdown
**Sarah** — First-time customer. Shopping on mobile during a commute.
Expects checkout to take under 60 seconds. Low patience for errors.
```

Named personas force specificity. "A customer" invites hand-waving. "Sarah, shopping on mobile during a commute" forces the spec to answer "what happens on a slow 3G connection?"

### 2. Environment (WHERE)

Deployment mode, hardware, network, access method. Different environments exercise different spec paths.

```markdown
- **Client:** Mobile browser (iOS Safari, 3G connection)
- **Backend:** Microservices (auth, payments, inventory, orders, notifications)
- **Scale:** Black Friday traffic — 50x normal load
```

### 3. Goal (WHAT)

A single sentence in the persona's own words. Use a blockquote.

```markdown
> "I want to buy these 3 items, pay with my credit card, and get a
> confirmation email within a minute."
```

### 4. Scenario Steps (HOW)

5-8 concrete steps the persona takes. Each step names:

- **The user action** — what they do
- **The primitives exercised** — which spec concepts activate
- **Gap detection questions** — 2-3 questions the simulator must answer

```markdown
#### Step 3: Payment fails, customer retries

Sarah's first payment attempt is declined. She re-enters a different card.

**Primitives:**
- `payments-spec.md`: retry policy, idempotency keys
- `inventory-spec.md`: stock hold duration during retry
- `orders-spec.md`: order state transitions on payment failure

**Questions:**
- Q3.1: The payment spec says "retry 3 times." The inventory spec
  holds stock for 5 minutes. What if retries take longer than 5 minutes?
- Q3.2: Does the order stay in "pending_payment" during retries, or
  does it transition to "failed" and require a new order?
```

**Rules for good steps:**

- Each step must cite at least one spec doc
- Each step must ask at least one question the spec should answer
- Questions use `Q<step>.<number>:` format for traceability
- Questions must be spec-answerable (yes/no/how), not opinion questions

### 5. Spec Coverage Matrix (COVERAGE)

A table showing which spec docs were exercised at which steps.

```markdown
| Spec Doc | Steps Hit | Coverage |
|----------|-----------|----------|
| `payments-spec.md` | 3,4 | Retry covered; hold-vs-retry timing gap |
| `inventory-spec.md` | 2,3 | Stock hold covered; expiry-during-retry unclear |
| `shipping-spec.md` | — | Not exercised |
```

Specs that no scenario touches are untested blind spots.

### 6. Gap Detection Questions Summary

Collect all Q-numbers for easy reference. The simulator answers every one.

From knot0-com/vibe-testing, MIT.