agentic-eval

Files (6)

• SKILL.md
• openai.yaml · agents/
• eval-checklist.md · assets/
• evals.json · evals/
• trigger-queries.json · evals/
• patterns.md · references/

SKILL.md (first lines)

# Agentic Evaluation

Use this skill when you are designing or implementing an evaluation loop that lets an agent assess and improve its own outputs through iteration — not when you are running a pre-existing test suite or doing a one-off review with no refinement cycle.

The core pattern is: **Generate → Evaluate → Critique → Refine → Output**, looping until a convergence condition is met or a max-iteration budget is exhausted.

## Use this skill when

- Implementing a self-critique or reflection loop that feeds output quality back into generation.
- Building an evaluator-optimizer pipeline that separates generation from evaluation responsibilities.
- Designing LLM-as-judge scoring to compare or rank multiple candidate outputs.
- Adding rubric-based scoring with weighted dimensions to iterative generation.
- Setting iteration limits, convergence checks, or structured evaluation output contracts.
- The task requires measurable improvement across runs, not just a single-shot best effort.

## Do not use this skill when

- You are running an existing test suite to verify code — use `verification-before-completion`.
- You are diagnosing a specific failure or bug, not evaluating output quality — use `systematic-debugging`.
- The goal is writing test coverage (unit tests, integration tests) — use `test-driven-development`.
- You are reviewing a completed artifact once without a refinement loop (a single code review, an editorial pass, a PR check).

## Routing boundary

| Situation | Use this skill? | Route instead |
| --- | --- | --- |
| Designing a reflection loop with a score threshold and max iterations | Yes | — |
| Implementing LLM-as-judge comparison of two candidate outputs | Yes | — |
| Running `npm test` to confirm a fix works | No | `verification-before-completion` |
| Tracing why a specific assertion fails | No | `systematic-debugging` |
| Writing Jest or pytest test coverage for a module | No | `test-driven-development` |
| Reviewing a PR diff once, no iteration | No | [`implementation-review`](../implementation-review/SKILL.md) |

## Inputs to gather

**Required before starting**

- The skill or agent behavior to evaluate.
- The evaluation objective and target metric for the application's outputs.

**Helpful if present**

- Existing evaluation fixtures or application quality criteria to extend.

## First move

1. Identify the application behavior to evaluate and the desired outcome.
2. Define the evaluation criteria and target metric.
3. Open the relevant reference file based on the evaluation type.

## Navigation

The three evaluation strategy patterns (outcome-based, LLM-as-judge, rubric-based) and full Python examples are in `references/patterns.md`.

The implementation checklist — criteria, threshold, loop wiring, convergence, logging — is in `assets/eval-checklist.md`.

For a new implementation, start with the checklist to confirm your setup is complete, then use the patterns reference to choose and adapt an evaluation strategy.

## Outputs

- Evaluation loop design with defined criteria, convergence check, and max iteration budget.
- Structured evaluation scores per iteration with input, output, and critique logged.
- Convergence or budget-exhaustion result confirming the loop terminated cleanly.

## Workflow

1. Name the application behavior under evaluation and the success criteria.
2. Choose a strategy from [`references/patterns.md`](references/patterns.md): outcome-based, LLM-as-judge, or rubric-based.
3. Complete the setup checklist in [`assets/eval-checklist.md`](assets/eval-checklist.md) (criteria, threshold, max iterations, logging).
4. Isolate generate → evaluate → optimize steps; keep the evaluator replaceable.
5. Run the loop with a hard `max_iterations` bound and a convergence check.
6. Log full trajectories (input, output, score, critique) for every iteration.
7. Stop on threshold met, no improvement, or budget exhaustion; report structured scores.
8. Feed failures back into evaluator criteria, prompts, or optimization steps, then re-run.

## Guardrails

- Always set a `max_iterations` bound (3–5 is a safe default) before wiring up a refinement loop. Unbounded loops stall agents.
- Require structured output (JSON) from the evaluation step so the optimize step has a reliable signal to act on. Free-text critique is fragile.
- Add a convergence check: if the score does not improve between iterations, stop early. Oscillating loops that never converge waste budget.
- Log the full iteration trajectory. Evaluation loops are hard to debug post-hoc without a history of inputs, outputs, scores, and critiques.
- Define evaluation criteria before generating any output. Criteria added mid-loop drift and make scores incomparable across iterations.
- Keep the evaluate step isolated from the generate step. Blending them makes it hard to replace the evaluator or diagnose score instability.
- Handle evaluation parse failures gracefully — if the LLM judge returns malformed JSON, fall back to a safe default (treat as failing) rather than crashing the loop.

## Validation

After implementing an evaluation loop, confirm:

- [ ] `max_iterations` is set and respected by the loop
- [ ] Evaluate step returns structured output (JSON or equivalent)
- [ ] Convergence check exits early when score does not improve
- [ ] All iterations are logged with input, output, score, and critique
- [ ] Parse-failure fallback is present on the evaluate step
- [ ] Criteria are defined before any generation begins

## Examples

- "Add a self-critique loop to my report-generation agent that retries up to three times if the rubric score is below 0.8."
- "Implement an evaluator-optimizer where a separate LLM judge scores code clarity and the generator rewrites until it passes."
- "Build a rubric-based evaluator with accuracy, completeness, and style dimensions that returns a weighted score as JSON."

## Reference files

- `references/patterns.md` — The three evaluation strategy patterns (outcome-based, LLM-as-judge, rubric-based) with annotated Python examples and a best-practices table.
- `assets/eval-checklist.md` — Implementation checklist: setup, loop wiring, convergence, logging, and safety items to confirm before shipping.

From matt-riley/agent-skills, GPL-3.0.

More from matt-riley/agent-skills