ai-test-generation

Files (2)

• SKILL.md
• prompt-patterns.md · references/

SKILL.md (first lines)

<objective>
A staged pipeline for generating tests with LLMs. The pipeline enforces structured intermediates at every step so that agents produce traceable, reviewable, high-quality tests instead of ad-hoc code.

**Before starting:** Check for `.agents/qa-project-context.md` in the project root. It contains tech stack, test frameworks, naming conventions, and project-specific patterns that dramatically improve generated test quality.
</objective>

---

## Discovery Questions

Before generating tests, clarify:

1. **What is the input source?**
   - PRD / feature spec / design doc
   - User story with acceptance criteria
   - Code diff (PR or commit range)
   - Bug report / incident report
   - API schema (OpenAPI / GraphQL SDL)

2. **What is the target test framework?**
   - **E2E:** Playwright (preferred), Cypress
   - **Unit:** Jest, Vitest, pytest
   - **API:** Playwright API testing, supertest, requests
   - **AI/LLM features:** Promptfoo, DeepEval, Ragas, or Braintrust — generate eval datasets, not Playwright specs (cross-link `ai-system-testing`)

2a. **What agent integration mode?** (Playwright projects only)
   - **Playwright CLI + SKILLS** (recommended for Claude Code / Codex / Cursor): token-efficient, runs inside the agent's loop. `npx playwright init-agents --loop=claude` scaffolds the planner/generator/healer.
   - **Playwright MCP** (`@playwright/mcp@latest`): higher overhead but right when the agent needs to *drive* a live browser interactively over a long-running session.
   - **Neither** — hand-write tests using AI as a scratch-pad helper.

3. **What project context is available?**
   - Existing test patterns to match?
   - Page Object Models or test helpers?
   - Data factories or fixtures?
   - CI constraints (timeout, parallelism)?

4. **What is the review workflow?**
   - AI generates full pipeline artifacts -> human reviews -> merge (default)
   - AI generates scenarios only -> human writes code
   - AI generates code -> human refines iteratively

5. **What domain knowledge is needed?**
   - Regulated industry (healthcare, finance) with compliance requirements?
   - Domain-specific invariants (money never goes negative, appointments cannot overlap)?
   - Known risk areas from past incidents?

---

## Core Principles

1. **Pipeline before code.** Never generate test code before establishing what to test, why, and how to verify it. The seven-step pipeline exists to prevent premature code generation.

2. **Structured intermediates are the product.** The assumptions document, coverage matrix, and oracle definitions are more valuable than the test code itself. They are reviewable, traceable, and reusable.

3. **Separate what from how.** Scenario generation (what to test) and oracle design (how to verify) are distinct cognitive tasks. Mixing them produces weak scenarios with weak assertions.

4. **AI generates first draft; human reviews and refines.** Never ship AI-generated tests without human review. The AI accelerates -- it does not replace judgment.

5. **Context is everything.** Feed the LLM your project conventions, existing test patterns, selector strategies, and data setup patterns. The more context, the less cleanup.

6. **Quality over quantity.** More tests is not better. Each test has a maintenance cost. Focus on critical paths, complex logic, and known risk areas.

---

## The Pipeline

**Mandatory workflow -- agents MUST follow this order:**

```
Step 1: Extract   → Requirements, entities, business rules from input
Step 2: Analyze   → Risks, invariants, edge cases, ambiguities
Step 3: Map       → Coverage matrix (requirement → scenario → priority)
Step 4: Generate  → Candidate scenarios (happy + boundary + negative + security + a11y)
Step 5: Design    → Assertions and oracles SEPARATELY from scenarios
Step 6: Code      → Test code (only after all above exist)
Step 7: Review    → Human review with traceability back to source
```

### Step 1: Extract Requirements and Entities

Parse the input source (PRD, story, diff, schema, bug report) into structured elements.

**Output: Requirements Document**

```markdown
## Source: [PRD name / Story ID / PR #123 / Bug #456]

### Entities
- User (roles: admin, member, guest)
- Order (states: draft, confirmed, shipped, delivered, cancelled)
- Product (attributes: name, price, stock, category)

### Business Rules
1. Only admin users can cancel orders after shipment
2. Stock must be decremented at order confirmation, not at checkout start
3. Free shipping applies when order total exceeds $99 before tax

### Explicit Requirements
- [REQ-1] User can add items to cart
- [REQ-2] User can proceed to checkout with valid payment
- [REQ-3] Order confirmation email is sent within 30 seconds

### Implicit Requirements (inferred)
- [IMP-1] Cart persists across browser sessions (assumed from "seamless experience" language)
- [IMP-2] Concurrent cart modifications resolve without data loss
```

**Key rule:** Separate explicit requirements (stated in source) from implicit requirements (inferred). Flag implicit ones for human confirmation.

### Step 2: Risk Analysis and Invariants

Derive what can go wrong, what must always be true, and where ambiguities exist.

**Output: Risk and Invariants Document**

```markdown
## Risks
| Risk | Likelihood | Impact | Source Requirement |
|------|-----------|--------|-------------------|

From JaiminVadadoriya/Distributed-Cloud-Storage---Sync-Platform, MIT.

More from JaiminVadadoriya/Distributed-Cloud-Storage---Sync-Platform