devops-engineer

Files (6)

• SKILL.md
• openai.yaml · agents/
• evals.json · evals/
• mcp-browser.md
• worked-examples.md · references/
• terraform.md

SKILL.md (first lines)

# DevOps Engineer — Test Pyramid Cascade

> Worked examples reference [DevCell](https://github.com/devcell-sh/devcell) paths and modules (`managed-mcp.nix`, `patchright-mcp-cell`, `/opt/npm-tools`). Adapt to your container setup.

## Overview

Shift-left testing for infrastructure. Find the cheapest way to get confidence in a change before spending expensive cycles (image rebuilds, container restarts, MCP reconnections).

**Core principle:** Every test decision is a trade-off between three dimensions — **Cost**, **Level**, and **Scope**. Navigate the pyramid from cheap to expensive, where each level inherits proven baselines from below.

## When to Use

- Infrastructure change needs validation (nix package, Dockerfile, MCP config, terraform)
- Existing E2E test is too expensive for the iteration cycle
- You're about to `task image:build` — stop and ask: can I validate this cheaper?
- A test is failing and you need to isolate which layer is broken
- Adding a new component that will integrate into an existing pipeline
- **User asks to run or test something** — discover available tests, classify by level/cost, present the pyramid table BEFORE executing. Let the user pick the level.

## When NOT to Use

- Application-level business logic testing (use standard TDD)
- One-off manual verification you won't repeat

## The Three Dimensions

| Dimension | Question | Example |
|-----------|----------|---------|
| **Cost** | Time/resources per iteration? | `node bot-detect-test.js` = 30s vs `task image:build` = 30min |
| **Level** | Where on the pyramid? How close to production? | L1 (isolated) = fast, narrow; L3 (pipeline) = slow, full fidelity |
| **Scope** | What's inside vs outside this test's concern? | "Proves bot detection. Does NOT prove nix integration or image build." |

**Always ask:** Is it cheaper to run the isolated test 100 times or the full build 100 times?

## Core Process

```dot
digraph cascade {
  rankdir=TB;
  "Topology mapped?" -> "Map components, refs, fallbacks" [label="no"];
  "Map components, refs, fallbacks" -> "Identify contract surface";
  "Topology mapped?" -> "Identify contract surface" [label="yes"];
  "Identify contract surface" -> "Extract cheapest test";
  "Extract cheapest test" -> "Document boundary";
  "Document boundary" -> "Present trade-off to user";
  "Present trade-off to user" -> "User accepts?" [shape=diamond];
  "User accepts?" -> "Run cheap test" [label="yes"];
  "User accepts?" -> "Adjust boundary" [label="no, too many assumptions"];
  "Adjust boundary" -> "Document boundary";
  "Run cheap test" -> "Pass?";
  "Pass?" -> "Level-Up needed?" [label="yes"];
  "Pass?" -> "Fix at this level" [label="no"];
  "Level-Up needed?" -> "Cascade up: cheap results become baseline" [label="yes"];
  "Level-Up needed?" -> "Done — scope sufficient" [label="no"];
  "Cascade up: cheap results become baseline" -> "Next pyramid level";
}
```

### Step 0: Announce the Pyramid Analysis

**Trigger:** This step runs **before any test execution** — including when the user asks to "run tests", "test X", "verify Y", or "check if Z works". Do not execute tests without presenting the pyramid first.

**What to do:**
1. **Discover** all available tests for the concern (grep test files, check `test/`, `test/js/`, Go test functions)
2. **Classify** each test by level (L1/L2/L3) and estimate cost
3. **Present** the full table to the user so they can pick the right level
4. **Wait** for the user to choose before running anything

**Level definitions:**

Every level has at least two steps: **gate** (static validation — "does it even parse?") then **execute** (run the actual test). If the gate fails, don't waste cycles on execution.

| Level | Name | Gate (static) | Execute | Cost profile |
|-------|------|--------------|---------|-------------|
| L1 | Isolated execution | Syntax check the source (`node --check`, `bash -n`, lint) | Run the component directly using the contract surface | Seconds to minutes |
| L2 | Build system | Parse/validate build config (`nix-instantiate --parse`, `tofu validate`) | Build the artifact, verify packaging/paths | Minutes |
| L3 | Full pipeline | Lint Dockerfile, validate compose config | Image build + container start + runtime validation | 10-30+ min |

**Announce format:**

```
Applying test pyramid analysis for: [change description]

Available tests:
| Level | Test | Command | Cost | What it proves |
|-------|------|---------|------|----------------|
| L1 | [name] | [command] | [time] | [scope] |
| L1b | [name] | [command] | [time] | [scope] |
| L2 | [name] | [command] | [time] | [scope] |
| L3 | [name] | [command] | [time] | [scope] |

Recommendation: Start at L[N] because [reason].
```

**Not all levels apply to every change.** Some changes only have L1 and L3 (no meaningful L2). Be explicit about which levels exist and which are skipped.

**Separate pyramids for separate concerns.** If the change spans multiple domains with different infrastructure requirements, each domain gets its own pyramid. For example:
- **Pyramid A (browser stealth):** L1=bot-detect-test.js → L2=nix build wrapper → L3=full image
- **Pyramid B (system libraries):** L1=nix build profile → L2=docker image layer → L3=full image

These pyramids share L3 (both converge at the full image build) but have completely different L1-L2 tests with different contract surfaces and scope limitations. Do not conflate them — test each concern through its own pyramid.

### Step 1: Identify the Contract Surface

Find the minimal shared interface between the component under test and its integration point. This is the seam where you can extract a test.

**Example — MCP bot detection:**
- Full pipeline: nix builds `patchright-mcp-cell` wrapper -> image includes it -> MCP server starts -> chromium launches with flags -> init-script injected -> page renders
- Contract surface: `chromium --use-gl=angle --use-angle=swiftshader ... + init-script content`
- If the isolated test uses the **exact same** launch args and init-script, results transfer to the integrated system

**How to find it:** Trace the call chain from the E2E test inward until you hit the boundary where "our code" meets "their code" or "config" meets "runtime." That boundary is the contract surface.

### Step 2: Extract the Cheapest Test

Build a standalone script that exercises just the contract surface. Requirements:
- Uses identical parameters as the integrated system (same flags, same config)
- Runs without image rebuild, container restart, or MCP reconnection
- Produces structured, comparable output (JSON, exit codes)

From DimmKirr/skills, MIT.

More from DimmKirr/skills