check-skill

Files (1)

• SKILL.md

SKILL.md (first lines)

# Skill activation check

A skill only earns its keep if the model reaches for it at the right moment. That
decision is made almost entirely from the skill's frontmatter `description` — so
"does my skill trigger?" is a measurable question, and this is how you measure it:
build a labelled set of user requests, run a real agent against each one, and score
whether the skill was engaged.

The user's request is: `$ARGUMENTS`

## Step 1 — Locate the target skill

`$ARGUMENTS` may be a path to a `SKILL.md`, a path to the skill's **directory**, a
skill name, or empty.

- Empty: glob `.claude/skills/*/SKILL.md` and `**/skills/*/SKILL.md`. One match →
  use it. Several → list them and ask which. None → say so and stop.
- A directory: use the `SKILL.md` inside it.
- A name: find the matching skill directory.

The **bare skill name is the directory name** containing `SKILL.md`. Read the
frontmatter `description` and keep it in front of you: that string is what the model
matches against, so it is the primary input to row design and the thing you will end
up recommending edits to.

**Measure its length while you are there — two separate budgets truncate it, and either
one produces a low-recall result that looks exactly like bad wording.**

1. **Per-skill truncation.** `description` and `when_to_use` are concatenated and cut at a
   fixed character budget — **1,536 characters**, configurable via the
   `skillListingMaxDescChars` setting. Trigger text past the cutoff cannot affect
   activation at all, so it may as well not exist.
2. **The whole-listing budget, which matters more in exactly the repositories that run
   activation suites.** The listing always contains every skill *name*, but its total
   character budget scales at about **1% of the model's context window**, shared across
   **every** skill the user has installed. When it overflows, Claude Code drops
   descriptions **starting with the skills you invoke least**.

The second one has a consequence worth stating plainly: in a many-skill repository a skill
can score near-zero recall with a **perfectly good description**, because its description
was never in the listing. Rewriting the wording then fixes nothing. And the drop order is
least-invoked-first, so a *newly authored* skill — which is by definition rarely invoked,
and is exactly what someone runs this suite on — is the most likely victim. That is a
systematic bias against the skill under test.

Levers, if the listing is the problem: `skillListingBudgetFraction` (the 1% default), the
`SLASH_COMMAND_TOOL_CHAR_BUDGET` environment variable (a fixed character count), and
`skillOverrides` set to `"name-only"` to free budget from skills you do not need matched.

If the skill has **no `description`** in its frontmatter, stop and report that as the
finding — a skill with no description can never be model-invoked, so a suite would
score zero recall by construction and tell you nothing you don't already know.

## Step 2 — Confirm coder-eval is installed

Run `coder-eval --version`. Installing this plugin did not install the CLI, and the
suite cannot be validated or run without it.

If it is missing, follow `${CLAUDE_PLUGIN_ROOT}/reference/cli-setup.md`: offer the
install, **ask before running it**, and confirm with `coder-eval --version`
afterwards. Never install unprompted, and do not continue if the user declines.

That reference also covers the other half of the version check — whether this project
pins a coder-eval version, and what to do when the installed one does not match it.

## Step 3 — Check what already covers this skill

Before designing anything, look at what the repository already has. Locate the task tree
per `${CLAUDE_PLUGIN_ROOT}/reference/repo-layout.md` and glob it for a task carrying a
`skill_triggered` criterion.

**If an existing suite already names this skill**, report its coverage — how many
positive and distractor rows, which `suite_thresholds` it gates on — and **offer to
extend it** by appending rows to its dataset, rather than scaffolding a parallel suite.
Two suites for one skill drift apart and the user pays for both on every run. Scaffold
only when nothing covers this skill, or when the user chooses to after seeing what is
there.

Two things worth reporting rather than silently working around:

- **An existing suite with no distractor rows.** Precision is 1.0 by definition, so half
  its result is meaningless. Offer to add distractors — a real improvement over both
  scaffolding a rival suite and saying nothing.
- **A neighbouring suite that names a *different* skill.** That is not coverage. Go ahead
  and scaffold, and say why the neighbour did not count.

**Then check where `agent.plugins` should come from.** If an experiment the task will
resolve against already supplies that block, **inherit it and do not write one** — a task
that redeclares what the experiment provides drifts from it, with nothing to catch the
divergence. Write the template's own block only when nothing already exposes the skill to
the sandbox. If several experiments exist and it is unclear which one this task resolves
against, ask rather than guess.

## Step 4 — Design the rows

This step is the whole experiment. The rest is mechanics.

**Positive rows** — requests a real user would plausibly make that the description
claims to cover. Paraphrase; never copy phrasing out of the description. A row lifted
from the description tests string matching, not activation. Vary the vocabulary and
include at least one oblique row where the user describes their *problem* rather than
the operation the skill performs.

**Distractor rows** — adjacent requests the skill should *not* claim, especially ones
that share vocabulary with the description. These are what make precision meaningful.

**Sibling-owned rows** *(optional)* — requests that legitimately belong to a **named
other skill in the same repository**, with `expected_skill` set to that sibling. In a
multi-skill repository, misrouting between two adjacent skills is the common failure, and
a plain distractor only shows *that* a misfire happened, not *where it went*. These rows
say where.

Add them when two skills have overlapping subject matter and you want to know which one
wins. They cost extra rows, and **every row is a full agent run** — so treat them as a
targeted follow-up, not a default.

**Never name the skill in a prompt.** That tests obedience, not activation:

- Bad: "Use the pdf-forms skill to fill in this application."

From UiPath/coder_eval, Apache-2.0.

More from UiPath/coder_eval