agentclash-challenge-pack-planner

Files (1)

• SKILL.md

SKILL.md (first lines)

# AgentClash Challenge Pack Planner

## Purpose
Turn an eval idea into a concrete challenge-pack plan before anyone writes YAML.

Use this skill to produce the planning artifact that downstream skills can convert into an AgentClash challenge pack. Planning does not require CLI access, but the plan must match the current challenge-pack model: `pack`, `version`, optional top-level `tools`, `challenges`, and `input_sets`.

## Use When
- A user describes a benchmark, regression idea, eval suite, or "can we test this agent?" scenario.
- The workload needs boundaries before YAML authoring: what behavior is tested, what cases exist, what counts as success, and which tools/files are allowed.
- A coding agent needs enough AgentClash product context to plan a pack without reading the AgentClash source repo.
- You need a handoff to `agentclash-challenge-pack-yaml-author`, `agentclash-challenge-pack-input-sets`, `agentclash-challenge-pack-scoring-validators`, or `agentclash-challenge-pack-llm-judges`.

## Do Not Use When
- The user already has a finished plan and wants valid YAML; use `agentclash-challenge-pack-yaml-author`.
- The task is only to validate or publish a YAML file; use `agentclash-challenge-pack-validation-publish`.
- The task is to choose deployments or start runs; use `agentclash-agent-deployment-setup` or `agentclash-eval-runner`.
- The user needs CLI installation, auth, workspace linking, or hosted setup; use `agentclash-cli-setup`.

## Inputs Needed
- Evaluation goal: the behavior, capability, or failure mode the pack should expose.
- Target agent class: coding agent, support bot, research agent, workflow agent, extraction agent, etc.
- Good, bad, and borderline outputs.
- Expected evidence: final text, JSON fields, captured files/directories, artifacts, metrics, latency, cost, or judge rationale.
- Case inventory: representative, edge, adversarial, regression, and smoke examples.
- Execution needs: `prompt_eval` versus `native`, tools, network, files, packages, secrets, and time budget.
- Release intent: exploration, regression suite, CI gate, public comparison, or customer demo.

## Planning Procedure
1. State the pack boundary in one sentence: what is being tested and what is explicitly out of scope.
2. Choose execution mode:
   - `prompt_eval` for prompt-style tasks that do not need pack-defined tools, sandbox config, or native file/tool execution.
   - `native` when the agent must use files, tools, sandbox policy, network, packages, artifacts, or code/file validators.
3. Define one or more `challenges`. Each challenge should have a stable `key`, title, category, difficulty (`easy`, `medium`, `hard`, or `expert`), and instructions.
4. Design cases before scoring. For each case, define a stable `case_key`, the `challenge_key` it targets, concrete inputs, expected outputs or expectations, and why the case exists.
5. Group cases into `input_sets`: at minimum `smoke` or `default`; add `full`, `regression`, or `ci` only when their purpose and budget differ. Each input set must contain cases for a single `challenge_key`; split mixed-challenge suites into separate input sets.
6. Pick evidence sources. Decide whether success is visible in `final_output`, structured JSON, files, artifacts, tool behavior, metrics, or LLM-judge rationale.
7. Choose scoring:
   - deterministic validators for exact, regex, JSON, numeric, token overlap, math, file, directory, or code-execution checks.
   - LLM judges for subjective quality where deterministic checks cannot honestly capture the behavior.
   - hybrid scoring when hard gates and qualitative judgment both matter.
8. Decide runtime policy only if needed: allowed tool kinds, sandbox network access, package needs, file assets, and secrets. Keep the policy as narrow as the workload allows.
9. Define publish criteria: what must be true before the pack can be validated, published, and used in an eval run.
10. Produce a handoff plan naming the next skill and the missing information, if any.

## Challenge Pack Model
Use these product nouns consistently:

- `pack`: human metadata such as `slug`, `name`, `family`, and optional description.
- `version`: executable version data: `number`, `execution_mode`, `evaluation_spec`, and optional `tool_policy`, `filesystem`, `sandbox`, and `assets`.
- `tools`: optional top-level pack-defined composed tools. Do not plan these for `prompt_eval`.
- `challenges`: task definitions. Cases reference them by `challenge_key`.
- `input_sets`: named groups of runnable cases.
- `cases`: concrete workload items with `case_key`, `payload`, `inputs`, `expectations`, case `artifacts`, and case-local `assets` as needed.
- `evaluation_spec`: score contract with `judge_mode`, validators, optional metrics or LLM judges, runtime limits, pricing, and scorecard dimensions.

## Planning Heuristics
- Prefer fewer, sharper challenges over a broad pack that mixes unrelated behaviors.
- Prefer small smoke sets that fail fast and full sets that measure coverage.
- Each case should teach the evaluator something unique. Duplicate cases need a reason, such as variance or regression coverage.
- Make expectations observable. "Looks good" is not enough; specify the evidence path and what a pass means.
- Use deterministic validators for hard facts, schemas, files, and code behavior. Use LLM judges for judgement calls like helpfulness, prioritization, style, tradeoff quality, or incident reasoning.
- Use `native` only when the task truly needs sandbox/files/tools. Simpler packs are easier to validate and reuse.
- Do not put raw secrets in the plan. Name the required secret keys and say they must be provided through workspace secrets or runtime/provider configuration.
- Do not invent IDs. Planning should name resources by role until validation/publish creates real IDs.

## Execution Mode Decision Table
| Need | Plan |
| --- | --- |
| Single prompt and final text answer | `prompt_eval` |
| Structured extraction from text input | `prompt_eval` unless file tooling is required |
| Agent must read/write files, run tests, or produce artifacts | `native` |
| Pack-defined custom tools | `native` |
| Network access or extra packages | `native` with explicit sandbox policy |
| Code, file, directory, or artifact-backed validators | `native` |
| Pure qualitative grading | `prompt_eval` or `native`, plus `llm_judge` depending on execution needs |

## Scoring Plan Shape
For each scoring dimension, specify:

```text
Dimension: <stable key>
Source: validators | metric | reliability | latency | cost | behavioral | llm_judge
Evidence: <final_output | file:path | artifact key | metric collector | judge key>
Pass rule: <threshold, gate, or qualitative rubric>
Failure message: <what should be reported when this fails>
```

Use `judge_mode: deterministic` when validators and metrics are sufficient. Use `judge_mode: llm_judge` when judges are the main grading surface. Use `judge_mode: hybrid` when deterministic gates and LLM-judge dimensions both matter.

## Case Coverage Checklist
- Happy path: the most ordinary success case.
- Edge case: unusual but valid input.
- Negative or guardrail case: input that should be rejected, abstained from, or handled safely.
- Ambiguity case: forces prioritization or asks for clarification when appropriate.
- Regression case: a known previous failure, with the evidence that should prevent recurrence.
- Budget case: confirms the pack can run within intended time, tool, and cost limits.

## Tool, Sandbox, And Artifact Planning
Only include these when the workload needs them.

- Allowed tool kinds in `version.tool_policy.allowed_tool_kinds` must use supported broad kinds such as `browser`, `build`, `data`, `file`, and `network`.
- `version.sandbox.network_access` should stay false unless the task needs outbound network.
- `version.sandbox.network_allowlist` should be specific when network is needed.
- `version.sandbox.additional_packages` should name only packages required by the workload or validators.
- Version, challenge, and case assets should have stable `key` and `path`; artifact-backed assets also need an `artifact_id` after upload.
- Case expectations can use `value`, `artifact_key`, or `source`. Supported `source` values are empty, `input:<case-input-key>`, or `artifact:<version-asset-key>`.

## Output Format
```text
Pack name:
Slug/family:
Goal:
Out of scope:
Target agent:
Execution mode: <prompt_eval | native>

Challenges:
- key:

From agentclash/agentclash, MIT.

More from agentclash/agentclash