skill-upper

Files (36)

• README.md
• README.zh.md
• SKILL.md
• case.yaml.tmpl · assets/
• eval.yaml.tmpl · assets/
• README.md · evals/
• add-case-to-existing.yaml · evals/cases/
• credential-presence-only.yaml · evals/cases/
• english-context-generates-english-only-cases.yaml · evals/cases/
• evolve-existing-skill.yaml · evals/cases/
• external-trace-review.yaml · evals/cases/
• plan-without-mutation.yaml · evals/cases/
• preserve-localized-regressions.yaml · evals/cases/
• scaffold-new-evals.yaml · evals/cases/
• scaffold-with-qodercli-engine.yaml · evals/cases/
• scaffold-with-script-judge.yaml · evals/cases/
• scaffold-with-user-simulator.yaml · evals/cases/
• eval-dashscope.yaml · evals/
• eval.yaml · evals/
• assert-credential-presence.sh · evals/fixtures/scripts/
• assert-english-only-generated-cases.sh · evals/fixtures/scripts/
• assert-localized-regressions.sh · evals/fixtures/scripts/
• assert-read-only-plan.sh · evals/fixtures/scripts/
• SKILL.md · evals/fixtures/skills/sample-no-evals/
• SKILL.md · evals/fixtures/skills/sample-with-evals/
• add-todo.yaml · evals/fixtures/skills/sample-with-evals/evals/cases/
• eval.yaml · evals/fixtures/skills/sample-with-evals/evals/
• requirements.txt · evals/
• test_regression_scripts.py · evals/
• case-yaml.md · references/
• cli.md · references/
• eval-yaml.md · references/
• install.md · references/
• judge-types.md · references/
• migrate-anthropic.md · references/
• observations.md · references/

SKILL.md (first lines)

# use-skill-up-cli

Help the user evaluate and evolve Agent Skills through the `skill-up` CLI.

Manual: <https://alibaba.github.io/skill-up/>

## Language Policy

**Default to English when responding to the user. If the user writes in Chinese (or any other language), switch to that language and stay consistent with the user's input throughout the session.**

Detection rules (highest priority first):

1. The user explicitly specifies a language in the current message (e.g. "answer in English" or "reply in Chinese") → follow the user's instruction.
2. The natural language used in the user's current message → match it.
3. None of the above → use English (default).

Regardless of the response language, technical identifiers in this SKILL — CLI commands, `eval.yaml` / `case.yaml` field names, report field names, etc. — MUST stay in their original English form. Do not translate them.

### Language Rules for Generated Artifacts

Use the user's current language for replies, newly authored explanatory prose,
and YAML comments. Preserve the language and semantics of test inputs, expected
output literals, identifiers, commands, paths, and existing regression cases.
Do not translate a deterministic assertion merely because the conversation
language changed. When testing localized behavior, keep the original business
text even in an English conversation.

Treat `assets/*.tmpl` as structural references: adapt placeholder prose and
comments to the requested language. Review only files created or changed for
this task; never rewrite unrelated cases to enforce a language preference.

Use short field-leading comments for generated YAML. Explain actual configured
fields: `schema_version`, `environment.type`, `engine.name`, `cases.files`, and
`report.formats` in eval.yaml; `id`, `title`, `input.prompt`, and `judge.type` in
case YAML. For nested fields, place the comment inside the parent mapping
immediately before the child field, rather than only before the parent.
For example, comment both `prompt` and `type` at their own indentation:

```yaml
input:
  # User request sent to the agent.
  prompt: Review this code.
# Cheap prerequisites checked before the judge.
expect:
  must_contain: ["suggestion"]
judge:
  # Deterministic grading strategy for this case.
  type: rule_based
```

Comment `expect` and a top-level `judge` when present; do not add an
optional field just to satisfy a comment check. Keep field names and enum values
unchanged. `skill-up import` does not preserve template comments.

## What is skill-up

`skill-up` is an evaluation CLI for Agent Skill authors. It installs the Skill into a real Agent Engine (Claude Code, Codex, qodercli, etc.), spins up an execution environment for each case, runs the prompt, then grades the result via declared rules / LLM judges / custom scripts, and finally produces a report.

Typical layout:

```
my-skill/
  SKILL.md
  evals/
    eval.yaml
    cases/
      <case-id>.yaml
    fixtures/
```

## When to trigger

Use this skill in any of the following situations:

- The user asks to "run / evaluate / verify / test this skill".
- The user asks to "fix / improve / iterate / evolve this skill" from eval failures.
- The user wants to "add evals, test cases, or regression cases to a skill".
- The user asks to record explicitly attributed Skill usage, evidence, or feedback.
- The user wants to review captured Skill observations or turn approved feedback into a regression case.
- The user wants to edit `eval.yaml` / `case.yaml`, or asks you to choose an appropriate `judge` type.
- The user mentions `skill-up run/validate/list-cases/report/import/init`.
- The user wants to migrate from Anthropic `evals.json` to skill-up.
- The current working directory contains `evals/eval.yaml` or `evals/evals.json` and the user wants to run it.

## Choose the requested workflow

- **Diagnose or plan:** read the target Skill and supplied evidence, identify
  concrete problems, and propose changes or cases. Do not install tools, create
  eval files, change configuration, or run evaluations unless requested.
- **Create or edit cases:** use Steps 1–4 to create or update the suite and
  validate its configuration. Stop before credentials and execution unless the
  user also requests a run. Report the actual config/case paths and validation
  status in the final reply.
- **Evaluate:** follow Steps 0–7 for the requested suite; report failures without
  automatically editing the Skill.
- **Improve:** follow the evaluation workflow and Step 8 within the requested
  scope. Preserve valid existing cases and assertions.

## Evaluation workflow

### Observation capture mode (optional)

Use this mode when the user asks to record the current Skill interaction or
feedback and observation tools are available. It requires either a current
Codex release with the `codex-skill-up` plugin and lifecycle-hook support, or DSH with
the skill-up plugin's observer explicitly enabled. Do not retrofit observation
capture onto the pinned Codex 0.80.0 evaluation adapter.

If the user asks how to install either host plugin, or the required observation
tools are missing and the user asks to enable them, read
`references/install.md` under "Install an observation host plugin (optional)". Do not
install or enable a plugin unless the user asks.

Claude Code, qodercli, Qwen Code, and other Agent Engines are not supported for
this observation workflow. This restriction does not apply to normal `skill-up`
evaluation runs, which remain multi-engine.

1. Attribute the interaction only when the user explicitly references a Skill
   (for example, `$my-skill`) or the responsible Skill is otherwise known with

From alibaba/skill-up, Apache-2.0.

More from alibaba/skill-up