task

Files (1)

• SKILL.md

SKILL.md (first lines)

# Author a coder-eval task

You are writing coder-eval task YAML. The user's request is: `$ARGUMENTS`

If `$ARGUMENTS` is empty, ask what the task should test. Do not invent a subject.

Good tasks use simple prompts: state the goal and the expected output, then let the
agent work out the approach. A single request can produce **several** task files —
"create tasks for all the registry subcommands" means one task per subcommand.

## Step 1 — Understand the request, and check the CLI is there

Run `coder-eval --version` first. Steps 6 and 7 both shell out to it, and finding that out
*after* writing several task files means the user gets a bare `command not found` with
nothing to act on. Installing this plugin did not install the CLI.

If it is missing, follow `${CLAUDE_PLUGIN_ROOT}/reference/cli-setup.md`: offer the install,
**ask before running it**, and confirm with `coder-eval --version` afterwards. Never install
unprompted, and do not write any task files if the user declines.

That reference also covers the other half of the version check — whether this project pins a
coder-eval version, and what to do when the installed one does not match it.

Then establish:

- **What is being tested** — which tool, SDK, CLI, skill, or capability?
- **How many tasks** — one operation, or several?
- **Difficulty** — smoke, basic, or intermediate?
- **Dependencies** — network, packages, starter files, external services?

State any assumptions you make rather than silently picking.

## Step 2 — Look at what already exists

Find the repository's task tree by following
`${CLAUDE_PLUGIN_ROOT}/reference/repo-layout.md`, and say what you resolved. If a task
already covers this ground, say so and offer to modify it instead of adding a
near-duplicate.

**Repo-local convention beats anything bundled with this plugin — where the two
disagree, the repo wins.** Before writing, read what the repository declares about task
authoring: its own contributor or convention documents, a task template if it ships one,
and a few neighbouring tasks. Adopt what you find — naming, tags, thresholds, weights,
where files go — and **say in your report which conventions you adopted**, so the choice
is visible rather than implied.

Two limits on that:

- **A repository that declares nothing** leaves the bundled rubric as the whole answer.
  Precedence is about deferring to a local rule that exists, not about doing nothing
  until one does.
- **Precedence covers style, not soundness.** If a local convention would produce a
  criterion that cannot fail, the rubric's correctness checks still bite — follow the
  convention where you can, say plainly where you did not and why.

## Step 3 — Design the task

**Task ID** — lowercase kebab-case, unique, `<domain>-<action>` (e.g.
`registry-list-processes`).

**Initial prompt** — minimal. State the goal and the expected output; nothing else.

- Good: "Use the `foo` CLI to list the available processes and save the result to
  `processes.json`."
- Bad: a step-by-step recipe with the exact flags, or a restatement of what the
  criteria check.

**Key rule: prompts instruct, criteria validate.** Never leak criteria detail into the
prompt. If a criterion checks that the output contains a `count` field, the prompt must
not mention `count` — otherwise you are testing transcription, not capability.

The subtle version of this, and the easiest to write by accident: a criterion that
matches a literal the prompt already dictates. "Use `pypdf` to read the fields" in the
prompt plus a criterion grepping for `pypdf` is a criterion that cannot fail — the agent
was told the answer. Either the constraint is a real requirement (keep it in the prompt,
and score what the agent *did with it* instead) or it is the thing under test (drop it
from the prompt). Never both.

(The rubric below carries this same trap as a review-time check, and is the declaration a
reviewer applies. The paragraphs above are the authoring-time version: they exist to stop you
writing it in the first place.)

**Success criteria** — read `${CLAUDE_PLUGIN_ROOT}/reference/task-rubric.md` *before*
choosing them. It is what this work will be checked against in step 5, and a criterion set
designed against it is far cheaper than one repaired after the fact.

Pick by what actually needs verifying:

| What to check | Criterion type |
| --- | --- |
| File exists, has content, matches a pattern | `file_check` (prefer over `file_exists` + `file_contains`) |
| JSON structure or specific values | `json_check` (JSON Schema + JMESPath assertions) |
| A script runs, tests pass, or a scorer emits a float | `run_command` |
| Output resembles a reference solution | `reference_comparison` |
| Subjective or open-ended quality | `llm_judge` |
| A deep, tool-using verdict on the sandbox | `agent_judge` (expensive) |
| The agent used a specific tool | `command_executed` |
| Tool-call efficiency against a budget | `commands_efficiency` |
| The agent engaged a target skill | `skill_triggered` (see `/coder-eval:check-skill`) |
| A predicted label vs. ground truth | `classification_match` |

Read `${CLAUDE_PLUGIN_ROOT}/reference/criteria.md` for each type's exact fields — it is
generated from coder-eval's own models, so it is the authoritative field list.

Rules that matter:

- **Every task needs at least one criterion that checks output *content***, not just
  existence. A suite of `file_exists` checks passes when the agent writes an empty file.
- Use `command_executed` sparingly — only when it genuinely matters *how* the result was
  produced. Set `require_success: true` whenever the command's success is what you are
  grading; the permissive default (`false`) counts a crashed invocation as evidence the
  work was done, and survives only for a genuine exception — a probe whose failure is an
  acceptable outcome.
- When the prompt genuinely must name a literal — a flag like `--json`, an output
  filename — a criterion matching that literal is a **smoke check**, not evidence: it
  only proves the agent typed back what it was told. Keep it if you like, at a low
  weight, and put the weight on a criterion that checks the resulting *behaviour*.
- Set `read_only: true` on a `run_command` grader that only inspects artifacts: a run
  whose agent crashed or timed out then still records what the artifacts were worth,

From UiPath/coder_eval, Apache-2.0.

More from UiPath/coder_eval