asc-ml-eval

Files (2)

• SKILL.md
• openai.yaml · agents/

SKILL.md (first lines)

# Asc Ml Eval

Evaluation is the control surface.
If you cannot say how the change is measured, the work is still too fuzzy.

## Workflow

### 1. Define the decision

State what the eval is trying to decide.

- Is the model good enough.
- Is one variant better than another.
- Did a change regress a slice.
- Is the system ready to ship.

### 2. Pick the right unit of measurement

- Use per-example checks when the task is local.
- Use grouped slices when failures cluster by domain, language, risk, or customer segment.
- Use end-to-end checks when the user experience matters more than turn-level accuracy.

### 3. Write the rubric

Keep the rubric aligned to the real task.

- correctness
- policy adherence
- required conventions
- helpfulness or preference, when relevant
- slice-specific failure checks

Prefer deterministic checks where the rule is lexical or structural.
Use a judge when the judgment is genuinely semantic.

### 4. Build the eval set

- Hold out examples that match the target workload.
- Include hard cases and common cases.
- Keep provenance clear.
- Separate training data from eval data cleanly.

### 5. Set the release gate

Define the threshold before you look at results.

- What must pass.
- What can degrade a little.
- What is an automatic stop.
- What needs human review.

### 6. Track regression over time

- Compare against the last shipped version, not just a single baseline.
- Watch slices, not only the average.
- Keep the eval reproducible so the same change can be checked again later.

## Output Shape

When you use this skill, answer with:

- the decision the eval is supporting
- the rubric
- the eval set shape
- the gate
- the failure slices that matter most

## Design Rule

If the task needs a new metric, ask whether a simpler slice or rubric already answers the question.
Do not invent a new score just because the old one is inconvenient.

From cavaunpeu/agent-skills-closet, MIT.

More from cavaunpeu/agent-skills-closet