agentclash-challenge-pack-llm-judges

Files (1)

• SKILL.md

SKILL.md (first lines)

# AgentClash Challenge Pack LLM Judges

## Purpose
Add LLM-as-judge scoring only where deterministic validators cannot capture the whole evaluation. Judges should complement objective checks, not replace them.

Use this skill after deterministic validators and evidence sources are known. A good judge is narrow, evidence-bound, budget-aware, and wired to one scorecard dimension through `source: llm_judge`.

## Use When
- Quality depends on reasoning, helpfulness, style, relevance, faithfulness, or nuanced task completion.
- A deterministic validator would be brittle or incomplete.
- A pack needs rubric, assertion, reference, or n-wise cross-agent ranking.
- The scorecard needs judge rationale, confidence, variance, sample count, and model count in replay/scorecard evidence.

## Do Not Use When
- The behavior can be scored with deterministic validators.
- Evidence sources are not stable yet; use input-sets, artifacts, and scoring validators first.
- The run cannot afford extra model calls.
- The judge would need secrets in prompt text or private data that should not leave the workspace.

## Environment
Use hosted production for CLI examples unless the user intentionally targets a local or self-hosted backend.

```bash
export AGENTCLASH_API_URL="https://api.agentclash.dev"
```

`agentclash challenge-pack validate` calls the hosted API and requires auth plus a workspace. Use `agentclash link`, `--workspace`, `AGENTCLASH_WORKSPACE`, or `.agentclash.yaml` before validating.

## Validation Commands
Validate after changing `judge_mode`, `llm_judges`, judge evidence, scorecard judge dimensions, consensus, or judge limits.

```bash
agentclash challenge-pack validate path/to/pack.yaml
agentclash challenge-pack validate path/to/pack.yaml --json
```

Human output prints `Challenge pack is valid` or `Challenge pack has errors`. Use `--json` for structured `valid` and `errors` fields.

## Minimal Hybrid Shape
Current evaluation specs still require at least one deterministic validator. Use `judge_mode: hybrid` when judges are paired with validators.

```yaml
version:
  evaluation_spec:
    name: support-quality
    version_number: 1
    judge_mode: hybrid
    validators:
      - key: mentions_policy
        type: contains
        target: final_output
        expected_from: literal:refund policy
    llm_judges:
      - mode: rubric
        key: helpfulness
        model: gpt-4o
        samples: 3
        context_from:
          - challenge_input
          - final_output
        rubric: |
          Score 1-5 for whether the answer is correct, complete, and easy for a support agent to use.
    scorecard:
      strategy: weighted
      dimensions:
        - key: correctness
          source: validators
          validators:
            - mentions_policy
          weight: 0.5
        - key: helpfulness
          source: llm_judge
          judge_key: helpfulness
          weight: 0.5
```

Mode coherence rules:

- `judge_mode: deterministic`: `llm_judges` must be empty.
- `judge_mode: llm_judge`: `llm_judges` must contain at least one judge.
- `judge_mode: hybrid`: `llm_judges` must contain at least one judge, and validators are still required by the current scoring validator.

## Judge Fields
Every `llm_judges[]` entry uses this source-backed shape:

```yaml
llm_judges:
  - mode: rubric
    key: stable_judge_key
    model: gpt-4o
    context_from:
      - final_output
    rubric: Rate the output 1-5.
```

Fields:

- `mode`: required; one of `rubric`, `assertion`, `reference`, or `n_wise`.
- `key`: required, unique, and must not collide with validator or metric keys.
- `model`: one model identifier.
- `models`: list of model identifiers. Set exactly one of `model` or `models`.
- `samples`: optional integer from `0` to `10`; `0` means default samples, currently `3`.
- `context_from`: optional list of supported evidence references.
- `output_schema`: optional JSON Schema draft-07 or 2020-12. Validation checks the schema; current judge message builders still instruct the built-in JSON response shape.
- `score_scale`: optional `{min, max}` for `rubric` and `reference`; `min` must be strictly less than `max`, default is `1..5`.
- `rubric`: required for `rubric` and `reference`.
- `assertion`: required for `assertion`.
- `expect`: optional boolean for `assertion`; when false, it flips the pass polarity.
- `prompt`: required for `n_wise`.
- `position_debiasing`: optional boolean for `n_wise`; rotates candidate order across samples.
- `reference_from`: required for `reference`; must be a supported evidence reference.
- `consensus`: required when `models` has more than one entry; invalid otherwise.
- `anti_gaming_clauses`: optional extra prompt clauses appended after the built-in judge instructions.
- `timeout_ms`: optional integer greater than `0`; default judge timeout is 60 seconds.

## Supported Evidence
Judge `context_from` and `reference_from` entries must use supported evidence references:

- `final_output`

From agentclash/agentclash, MIT.

More from agentclash/agentclash