agentclash-challenge-pack-yaml-author

Files (1)

• SKILL.md

SKILL.md (first lines)

# AgentClash Challenge Pack YAML Author

## Purpose
Write challenge-pack YAML that matches the current AgentClash parser and validators, without requiring access to the AgentClash source repo.

Use this skill after planning. The output should be a concrete YAML file plus the exact validation commands a coding agent should run before publish.

## Use When
- A challenge-pack plan needs to become valid YAML.
- An existing pack YAML needs source-compatible edits.
- A coding agent needs the exact YAML object shape for `pack`, `version`, `challenges`, `input_sets`, scoring, tools, sandbox, assets, cases, inputs, and expectations.
- The agent will later hand the file to `agentclash-challenge-pack-validation-publish`.

## Do Not Use When
- The user only has a vague eval idea; use `agentclash-challenge-pack-planner` first.
- The YAML is finished and only needs validation or publish; use `agentclash-challenge-pack-validation-publish`.
- The task is to start runs or choose deployments; use `agentclash-eval-runner` or deployment skills.

## Environment
Use hosted production unless the user intentionally points at another backend.

```bash
export AGENTCLASH_API_URL="https://api.agentclash.dev"
```

## CLI Commands
Start from the CLI template when possible, then edit the YAML.

```bash
agentclash challenge-pack init support-eval.yaml --template prompt_eval --name "Support Eval" --slug support-eval
agentclash challenge-pack init native-files.yaml --template native --name "Native Files" --slug native-files
agentclash challenge-pack validate support-eval.yaml
agentclash challenge-pack validate support-eval.yaml --json
```

Human validation output prints `Challenge pack is valid` or `Challenge pack has errors`. Use `--json` when a coding agent needs structured fields such as `valid` and `errors`.

## YAML Skeleton
These are the top-level fields accepted by the bundle parser:

```yaml
pack:
  slug: support-eval
  name: Support Eval
  family: support
  description: Evaluates concise customer support answers.

version:
  number: 1
  execution_mode: prompt_eval
  evaluation_spec:
    name: Support Eval Scoring
    version_number: 1
    judge_mode: deterministic
    validators:
      - key: mentions_refund_policy
        type: contains
        target: final_output
        expected_from: literal:refund policy
    scorecard:
      strategy: weighted
      dimensions:
        - key: correctness
          source: validators
          weight: 1

challenges:
  - key: refund-question
    title: Refund Policy Question
    category: support
    difficulty: easy
    instructions: Answer the customer in a concise, helpful tone.

input_sets:
  - key: smoke
    name: Smoke
    description: Fast validation cases.
    cases:
      - challenge_key: refund-question
        case_key: basic-refund
        payload:
          customer_message: Can I get a refund after 14 days?
        inputs:
          - key: prompt
            kind: text
            value: Can I get a refund after 14 days?
        expectations:
          - key: expected_policy
            kind: text
            source: input:prompt
```

## Required Fields
- `pack.slug`, `pack.name`, and `pack.family` are required.
- `version.number` must be greater than zero.
- `version.execution_mode` should be explicit: use `prompt_eval` or `native`.
- `version.evaluation_spec.validators` must contain at least one validator.
- Every challenge needs `key`, `title`, `category`, and `difficulty`.
- `difficulty` must be `easy`, `medium`, `hard`, or `expert`.
- Every input set needs `key`, `name`, and at least one `cases` entry.
- Every case needs `challenge_key` referencing a declared challenge and a stable `case_key`.
- Use `cases`, not legacy `items`, in new YAML.

## Execution Modes
Use `prompt_eval` for prompt-style tasks where the agent only needs the prompt and final output.

`prompt_eval` cannot use:
- top-level `tools`
- `version.tool_policy`
- `version.sandbox`

Use `native` when the challenge needs files, tools, network policy, package installation, file validators, directory checks, code execution, or sandbox behavior.

## Native Tools And Sandbox
Only include these blocks for `native` packs.

```yaml
tools:
  custom:

From agentclash/agentclash, MIT.

More from agentclash/agentclash