agentclash-dataset-workflows

Files (1)

• SKILL.md

SKILL.md (first lines)

# AgentClash Dataset Workflows

## Purpose
End-to-end dataset operations in a workspace: versioned example banks, eval runs against challenge packs, CI gating, synthetic generation, production trace import, and promotion into regression suites.

## Use When
- Building or curating labeled examples for prompt or agent evals.
- Gating merges on dataset eval pass rate vs a baseline.
- Importing OTEL/Braintrust/LangSmith/Phoenix/AgentClash traces as reviewable candidates.
- Syncing a dataset version into a linked regression suite.

## Do Not Use When
- The user only needs a one-off challenge-pack run — use `agentclash-eval-runner`.
- Prompt matrix experiments without a dataset artifact — use `agentclash-prompt-eval-playground`.
- Harness coding tasks — use `agentclash-agent-harness-setup`.

## Inputs Needed
- Workspace ID and dataset ID (create datasets via API/UI if none exist).
- For eval/gate: dataset version ID, challenge pack version ID, challenge key, deployment IDs.
- For gate: baseline ID and candidate run ID (or `--eval` to start eval inline).
- For generate: `--count`, `--provider-account`, `--model`.

## Environment
```bash
export AGENTCLASH_API_URL="https://api.agentclash.dev"
agentclash workspace use <WORKSPACE_ID>
agentclash dataset list
agentclash dataset view <DATASET_ID> --json
```

## Procedure
1. Inspect dataset and versions (`list`, `view`, `version list`).
2. Import or export examples; optionally create a version snapshot.
3. Run a dataset eval or attach an existing run.
4. Gate with `dataset test` against a baseline (CI-friendly `--format junit`).
5. Optionally generate synthetic examples, import traces, promote candidates, sync regression suite.

## Commands

### Inspect and mutate examples
```bash
agentclash dataset list
agentclash dataset view <dataset-id>
agentclash dataset version list <dataset-id>
agentclash dataset version create <dataset-id> --label "v2-seeds"
agentclash dataset import <dataset-id> examples.jsonl
agentclash dataset export <dataset-id> --version <version-id> > out.jsonl
agentclash dataset example list <dataset-id>
agentclash dataset example add <dataset-id> --input '{"messages":[...]}' --expected '{"score":1}'
agentclash dataset example edit <dataset-id> <example-id> --expected '{"score":1}'
agentclash dataset example rm <dataset-id> <example-id>
```

### Eval and CI gate
```bash
agentclash dataset eval <dataset-id> \
  --version <version-id> \
  --pack <pack-version-id> \
  --challenge <challenge-key> \
  --deployment <deployment-id>

agentclash dataset test <dataset-id> \
  --baseline <baseline-id> \
  --run <run-id> \
  --min-pass-rate 0.9 \
  --max-regressions 0 \
  --format junit

# Start eval then gate in one command
agentclash dataset test <dataset-id> \
  --eval \
  --version <version-id> \
  --pack <pack-version-id> \
  --challenge <challenge-key> \
  --deployment <deployment-id> \
  --baseline <baseline-id> \
  --timeout 30m
```

### Synthetic generation
```bash
agentclash dataset generate <dataset-id> \
  --count 50 \
  --provider-account <account-id> \
  --model <provider-model-id> \
  --create-version \
  --version-label "synthetic-v1" \
  --follow
```

### Trace import and promotion
```bash
agentclash dataset import-traces <dataset-id> traces.json --source otel
agentclash dataset import-traces <dataset-id> --source agentclash --run <run-id> --run-agent <run-agent-id>
agentclash dataset trace-candidates list <dataset-id> --status pending
agentclash dataset promote <dataset-id> <candidate-id> --tag production --expected '{"score":1}'
```

### Regression suite sync
```bash
agentclash dataset sync-regression-suite <dataset-id> \
  --version <version-id> \
  --pack <pack-version-id> \
  --challenge <challenge-key> \
  --suite-name "Dataset regression bank"
```

## Expected Output
- Eval creates a run; gate returns pass/fail with regression counts.
- `dataset test --format junit` exits 0 on pass, 1 on gate failure (422).
- Generate with `--follow` polls job until completion.

## Failure Modes
- Gate without `--baseline` → required.
- Gate without `--run` and without `--eval` → provide one.
- Generate missing provider/model → all three of count, provider-account, and model required.
- Sync regression without version/pack/challenge → all three flags required.

## Safety Notes

From agentclash/agentclash, MIT.

More from agentclash/agentclash