agentclash-scorecard-reader

Files (1)

• SKILL.md

SKILL.md (first lines)

# AgentClash Scorecard Reader

## Purpose
Turn completed or inspectable AgentClash run evidence into an engineering readout: who won, why, which claims are backed by scorecard/replay/artifact evidence, and what the next command or fix should be.

## Use When
- A user asks why an AgentClash run passed, failed, regressed, drifted, or picked a winner.
- You have a run ID and need to inspect rankings, run agents, scorecards, replay steps, failure-review items, or artifacts.
- A reviewer needs evidence-first findings instead of raw JSON dumps.
- A follow-up skill needs a grounded summary before promoting regressions or changing a challenge pack.

## Do Not Use When
- The user needs to start a new eval run; use `agentclash-eval-runner`.
- The user needs to author, validate, or publish the challenge pack first; use the challenge-pack skills.
- The user is ready to promote failures into regression suites; use `agentclash-regression-flywheel` after this skill identifies the useful failures.

## Inputs Needed
- Workspace ID or configured workspace context.
- Run ID.
- Optional run agent ID or agent label for a specific scorecard or replay.
- Optional baseline expectation, expected winner, or release-gate decision to compare against.
- Optional artifact IDs from failure evidence, scorecards, or workspace artifact list.

## Environment
Use hosted production by default unless the user intentionally targets local or self-hosted infrastructure:

```bash
export AGENTCLASH_API_URL="https://api.agentclash.dev"
agentclash auth status
agentclash workspace use <WORKSPACE_ID>
```

Workspace resolution follows the CLI setup rules: `--workspace`, `AGENTCLASH_WORKSPACE`, saved config, or `.agentclash.yaml`. `run failures`, `artifact list`, and `eval scorecard` require a workspace. `artifact download` uses an artifact ID directly.

## Procedure
1. Confirm the run exists and get its agent IDs.
2. Read the ranking to identify the winner, sort mode, gaps, unavailable scores, and evidence warnings.
3. Read the relevant scorecard. Use `eval scorecard` for run-first analysis and baseline comparison; use `run scorecard` when you already have a run agent ID.
4. Read failure-review items for concrete failed checks, replay refs, artifact refs, judge refs, metric refs, severity, and promotability.
5. Pull replay steps around referenced sequences, or page through replay when no refs exist.
6. Inspect artifacts only when a scorecard, failure item, or user request points to them.
7. Report claims as evidence-first findings: claim, evidence pointer, impact, next action.

## Commands
Start with run-level shape:

```bash
agentclash run get <RUN_ID> --json
agentclash run agents <RUN_ID> --json
agentclash run ranking <RUN_ID> --json
agentclash run ranking <RUN_ID> --sort-by composite --json
agentclash run ranking <RUN_ID> --sort-by correctness --json
```

Scorecard commands:

```bash
agentclash eval scorecard <RUN_ID> --agent <RUN_AGENT_ID_OR_LABEL> --json
agentclash eval scorecard --agent <RUN_AGENT_ID_OR_LABEL> --json
agentclash run scorecard <RUN_AGENT_ID> --json
```

Replay and failure-review commands:

```bash
agentclash run failures <RUN_ID> --json
agentclash run failures <RUN_ID> --agent <RUN_AGENT_ID> --severity blocking --json
agentclash run failures <RUN_ID> --class policy_violation --evidence-tier hosted_structured --json
agentclash run failures <RUN_ID> --cluster <FAILURE_CLUSTER_KEY> --limit 50 --json
agentclash replay get <RUN_AGENT_ID> --limit 50 --json
agentclash replay get <RUN_AGENT_ID> --cursor 50 --limit 50 --json
```

Artifact commands:

```bash
agentclash artifact list --json
agentclash artifact download <ARTIFACT_ID> --output <PATH>
```

Important exact CLI shapes:

- `run scorecard` takes one argument: `<RUN_AGENT_ID>`. It does not accept `<RUN_ID> <RUN_AGENT_ID>`.
- `replay get` takes one argument: `<RUN_AGENT_ID>`. It does not accept `<RUN_ID> <RUN_AGENT_ID>`.
- `artifact list` is workspace-wide. It does not have a `--run` filter today; use `artifact_refs`, `run_id`, or `run_agent_id` fields from JSON output to choose what to download.
- `run ranking --sort-by` commonly uses `composite`, `correctness`, `reliability`, `latency`, or `cost`. The backend also accepts a custom dimension key when that key exists in the scorecard dimensions; unknown keys return `invalid_sort_by`.
- `run failures --limit` defaults to 50 when omitted and is capped at 200 by the API.

## Ranking JSON
`agentclash run ranking <RUN_ID> --json` returns a stateful response:

```json
{
  "state": "ready",
  "ranking": {
    "run_id": "<RUN_ID>",
    "evaluation_spec_id": "<EVALUATION_SPEC_ID>",
    "sort": {
      "field": "correctness_then_reliability",
      "direction": "desc",
      "default_order": true
    },
    "winner": {
      "run_agent_id": "<RUN_AGENT_ID>",
      "strategy": "<strategy>",
      "status": "<status>",
      "reason_code": "<reason_code>"
    },
    "evidence_quality": {
      "missing_fields": [],
      "warnings": []
    },
    "items": [
      {
        "rank": 1,
        "run_agent_id": "<RUN_AGENT_ID>",
        "lane_index": 0,
        "label": "<agent label>",
        "status": "completed",

From agentclash/agentclash, MIT.

More from agentclash/agentclash