the-controller-validating-agent-applications

Files (1)

• SKILL.md

SKILL.md (first lines)

# Validating Agent Applications

## Overview

Testing agent applications without running agents is testing a shell, not an application.

**Core principle:** Every agent application must pass end-to-end validation that includes real agent execution across multiple scenarios, multiple times, with binary pass/fail semantic evaluation.

**Violating the letter of this rule is violating the spirit of this rule.**

**REQUIRED BACKGROUND:** You MUST understand the-controller-test-driven-development before using this skill. That skill defines the RED-GREEN-REFACTOR cycle. This skill applies it to agent application validation.

## The Iron Law

```
NO AGENT APPLICATION SHIPS WITHOUT PASSING E2E VALIDATION
```

If your tests don't run real agents, they're not E2E tests. If your E2E tests pass sometimes and fail sometimes, they're not tests — they're noise.

## When to Use

**Use when:**
- Building an application that calls LLM agents
- Modifying agent behavior, prompts, tool definitions, or orchestration logic
- Adding new agent capabilities or scenarios
- Before claiming an agent application works or is ready to ship

**Don't use for:**
- Pure UI testing with no agent involvement
- Unit testing deterministic code (use standard TDD)
- Testing skills themselves (use the-controller-writing-skills)

## The Three Phases

Every validation suite runs in this order. No exceptions.

```
DIAGNOSTICS → SCENARIOS → VERDICT
```

Skip diagnostics? You'll waste 10 minutes debugging a test failure that was actually an expired API key.

---

## Phase 1: Diagnostics (Pre-Flight)

**Purpose:** Eliminate environment failures before the first test runs. The main test path must never flake due to missing prerequisites.

**The diagnostics step runs before any scenario.** If it fails, no scenarios execute. This is a hard gate.

### Required Checks (in order)

| Check | How | Fail message |
|-------|-----|-------------|
| **Environment variables** | Assert each required var is present and non-empty | `Missing env var: ANTHROPIC_API_KEY` |
| **API key validity** | Lightweight auth call (list models, not a generation) | `API key invalid or expired` |
| **Model availability** | Verify the target model responds | `Model claude-haiku-4-5-20251001 not available` |
| **External services** | Ping databases, APIs the agent depends on | `Database at localhost:5432 unreachable` |
| **Rate limit headroom** | Check current usage against limits | `Rate limit: 90% consumed, tests may fail` |
| **File/data prerequisites** | Verify required fixtures, configs, data files exist | `Missing fixture: test-data/users.json` |

### Diagnostics Rules

- **Complete in under 5 seconds.** If your pre-flight takes longer, it's doing real work — move that to a scenario.
- **Fail with actionable messages.** Not "connection failed" but "Database at localhost:5432 unreachable — run `docker compose up -d` first."
- **Check everything.** If a scenario needs it, diagnostics should verify it.

```typescript
// Example: diagnostics function
async function runDiagnostics(): Promise<void> {
  // Hard requirements — fail immediately
  assertEnvVar('ANTHROPIC_API_KEY');
  assertEnvVar('DATABASE_URL');

  // Service availability — fail with actionable message
  await assertApiKeyValid(process.env.ANTHROPIC_API_KEY!, {
    failMessage: 'API key invalid — regenerate at console.anthropic.com'
  });

  await assertServiceReachable(process.env.DATABASE_URL!, {
    failMessage: 'Database unreachable — run: docker compose up -d'
  });

  // Rate limits — warn or fail based on headroom
  await assertRateLimitHeadroom({
    minimumRemaining: 100, // enough for test suite
    failMessage: 'Rate limit too low — wait or use a different key'
  });
}
```

---

## Phase 2: Scenarios

### Scenario Structure

Each scenario tests one specific agent behavior. Every scenario has:

```
Scenario: [name]
Input: [what the agent receives]
Expected:
  MUST: [required elements — ALL must be present to pass]
  MUST NOT: [forbidden elements — ANY present = fail]
  TOOL CALLS: [expected tool invocations, if applicable]
Pass condition: ALL MUST present AND zero MUST NOT present AND tool calls match
```

**There is no "partially correct." A scenario passes or it fails.**

### Scenario Coverage Taxonomy

Your test suite MUST include scenarios from each applicable category:

| Category | What it tests | Example |
|----------|--------------|---------|
| **Happy path** | Core expected behavior | Agent answers a straightforward question correctly |

From kwannoel/the-controller, MIT.

More from kwannoel/the-controller