workflow-from-eval

Files (5)

• SKILL.md
• gotchas.md
• ground-truth.md
• intake.md
• search.md

SKILL.md (first lines)

# Workflow from Eval

## When to use this skill

Prefer this skill when the user describes a business problem and wants a working solution,
but has not chosen a Workflow architecture, model, or approach. A goal plus images or video
is enough: help define success, build an approved Eval, and search for a suitable Workflow.
The user does not need to know this skill exists or ask for an Eval.

Also use it to improve an existing Workflow by comparing approaches against examples and
success criteria. Use direct tools for straightforward builds and edits with a clear approach;
use Workflow Evals tools directly to measure one existing Workflow once.

For example, "Here is a factory video; detect assembly mistakes for me" fits this skill.
"Add OCR after this crop" is a direct edit. For video, explain the per-frame scoring boundary
in the Goal Card; this skill does not validate stateful video execution or temporal events.

Introduce the approach in plain language: "Let's agree on examples of correct results, then
compare approaches to find a Workflow that works well for your goal." Follow the checkpoints
below before spending inference past them; recognizing the intent does not authorize an
unbounded search.

> **For agents:** This skill is authored in [`roboflow/computer-vision-skills`](https://github.com/roboflow/computer-vision-skills). If your client already has it loaded as a skill (for example from a Roboflow plugin), use that copy. The `skill://roboflow/<name>/SKILL.md` MCP resources serve the same skills for clients that don't have them loaded.

You are the optimizer. The user is the judge. They describe the job, look at examples, say what
is right and what is wrong, and say what they care about. You do everything else: build the
test suite and the scoring behind the scenes, compose a portfolio of very different candidate
Workflows, score them all together, tune the survivors, validate the winner on examples the
search never saw, and tidy up. The user never has to learn what a Spec, Case, BindingSet,
Execution, or subjectKey is. If they ask, explain; otherwise keep those words out of the
conversation.

The Workflow Evals API contract (Spec, Dataset, Cases, bindings, Runs, results) ships with
the engine: `workflow_evals_agent_manifest` lists its resources and
`workflow_evals_agent_resource_get` reads them (`skill:workflow-evals`, `skill:create-eval`).
Read them in preflight. This skill owns the protocol and the search; those resources own the
API contract.

Guides in this skill:

- `intake.md`: the questions, the Goal Card, how concerns become scoring.
- `ground-truth.md`: the test suite, getting examples from video, folders, projects
  or URLs, the contact sheets, labeling with the user's corrections, growing the suite.
- `search.md`: the portfolio, smoke, the joint Run, reading a round with its noise,
  tune-and-prune rounds, the finalists Run, cleanup, the report.
- `gotchas.md`: platform behaviors, known issues, and fallbacks that waste Runs.

When showing annotations, labels, or predictions over images or videos, load the installed
`draw-vision` skill (possibly named `roboflow:draw-vision`), or the bundled
[`../draw-vision/SKILL.md`](../draw-vision/SKILL.md). It owns library and environment choices;
load it only when rendering media evidence. If unavailable, use simple captioned media.

## Three principles that shape everything

**The test suite only grows.** Every example is a test somebody approved. Examples are added
or corrected, never removed. Every scoring Run uses the whole suite. Validation means adding
examples the search never saw and rerunning on the grown suite, not holding part of the suite
back. The suite outlives the session: a production miss becomes a new example, and any future
Workflow version is one Run away from a verdict.

**The Eval decides, not anyone's opinion.** Every candidate goes through the same scoring. No
claim about a candidate counts until a Run has scored it on the suite. Differences smaller
than the run-to-run noise you measured are ties, broken on the next criterion.

**The user judges from pictures.** Every message that asks "which of these are right" ships
the images with the proposed answer printed on each one (contact sheets or a grid in
`ground-truth.md`). A table of file names is not reviewable.

## Talk like this

| Say to the user | Means |
| --- | --- |
| the job, the goal | what the Workflow must answer per image |
| examples, the test suite | Cases in the Eval Dataset |
| answer key | ground truth |
| scorecard | the Spec's Checks, in words |
| must-never rule | a hard constraint that disqualifies a candidate |
| candidate | one Workflow under test |
| family | a group of candidates sharing an approach (VLM, detector plus logic, ...) |
| round | one joint Run scoring every live candidate on the whole suite |
| champion | the best candidate so far |
| new examples | Cases added after the search, for validation |

Speak the user's language (Portuguese if they write Portuguese) unless they ask for another;
the intake asks which language the answers, names and report should use, because output
names freeze after Phase 2. Keep every message to the user short: one table or one question
at a time, plus the images. Before every Run, say what it is and what it costs: "round 2:
14 candidates × 52 examples".

## Preflight

1. `workflow_evals_capabilities`. It returns `workspace` (the URL slug); if the user named a
   workspace and it differs, stop and tell them.
2. `workflow_evals_agent_manifest`, then read `skill:workflow-evals` and `skill:create-eval`
   with `workflow_evals_agent_resource_get`.
3. Pick a slug for the job (`shelf-gaps`) and a session timestamp `YYYYMMDD-HHMM`. The suite's
   resources are named by the job and reused across sessions: `<slug>-spec`, `<slug>-suite`,
   `<slug>-eval`. Only candidates and the working folder carry the session timestamp.
4. Look for an existing Eval for the job first (`workflow_evals_list`, name `<slug>-eval`). If
   it exists, reuse it, its Spec and its Dataset, and add examples; do not create a second
   suite for the same job. Listings carry signed URLs: read them, never paste them anywhere.
5. Create a working folder `./opt-<slug>-<timestamp>/` with `created.json`, a registry of every
   resource this session creates (kind, name, id, url slug, parent, pre-existing flag,
   pruned/deleted status). Pre-existing resources, including a reused Eval, go in with
   `pre-existing: true`. Cleanup reads this file and nothing else. The folder is also the
   resume point: `created.json`, `answer-key-draft.json`, `cases/all.json`, `candidates/*.json`
   and `rounds/*/summary.json` are written before every long wait, so a session that stops
   (laptop closed, context reset) continues from them.
6. Probe the model providers the portfolio will need with a one-image caption through
   `workflow_specs_run` (`search.md`, "Provider down"). Diagnose failures from the returned
   error, not how long they persist. Ask for a managed-key check or a user key only when
   the error indicates an authentication or key-configuration problem.

## The protocol

Five phases, four checkpoints. A checkpoint is a message where you show something and wait.
Never spend inference past a checkpoint before the user answers.

### Phase 1: understand the job

From roboflow/computer-vision-skills, Apache-2.0.

More from roboflow/computer-vision-skills