eval-add

Files (1)

• SKILL.md

SKILL.md (first lines)

# /sdlc:eval-add — every incident becomes an eval

Read `sdlc-loop`. Evals are the AI-native equivalent of stage-gate QA: a suite that runs whenever the agent's
configuration changes (CLAUDE.md, `.claude/**`, REVIEW.md, sdlc.config.json) and nightly.

## Build the case

1. Identify the source: an incident (`sdlc/incidents/<id>/intent.md` + the fixing PR), a PR, or a real task
   from recent work (`--from-task`). Read what the agent should have done differently.
2. Write `evals/cases/<kebab-name>.json`:
   ```json
   {
     "name": "status-endpoint-403-for-other-customer",
     "source": "sdlc/incidents/2026-07-03-status-leak",
     "prompt": "Add GET /claims/{id}/status behind existing auth per sdlc/changes/<id>/spec.md.",
     "setup": ["git checkout -q <base sha or fixture branch>"],
     "allowed_tools": "Read,Edit,Write,Grep,Glob,Bash(make test),Bash(make lint)",
     "max_turns": 40,
     "checks": [
       { "type": "command",      "run": "make test",  "expect_exit": 0 },
       { "type": "command",      "run": "make lint",  "expect_exit": 0 },
       { "type": "file_contains","path": "claims-api/routes/status.py", "pattern": "require_auth" },
       { "type": "no_changes",   "globs": ["tests/**", "src/gen/**"] },
       { "type": "output_contains", "pattern": "make test" }
     ]
   }
   ```
   Check types supported by `evals/check.sh`: `command`, `file_contains`, `file_not_contains`, `no_changes`,
   `output_contains`, `output_not_contains`.
3. The prompt must be realistic (what a person would actually type), and the checks must define "acceptable"
   deterministically: tests pass, lint clean, behaviour unchanged, policy followed.
4. Run it locally once: `evals/run.sh evals/cases/<name>.json`. It must pass with the current configuration
   (it is a regression test, not a wish).
5. Commit on a branch, PR titled `eval: <name>`, description links the incident/PR. The team that owned the
   incident owns the eval.

## Keep the suite alive

- As models improve, cases stop discriminating. Mark such cases `"retired": true` with a reason instead of
  deleting them; `run.sh` skips retired cases.
- Aim for 20–50 live cases drawn from real tasks. Report the current count and pass rate after adding.

From bibutikoley/ai-sdlc-claude, MIT.

More from bibutikoley/ai-sdlc-claude