lythoskill-arena

Files (11)

• SKILL.md
• COMMANDS.md · references/
• arena-runtime.md · references/
• configuration-schemas.md · references/
• design-principles.md · references/
• pareto-analysis.md · references/
• player-setup.md · references/
• reproduce-sh-bdd-contract.md · references/
• reproduce-sh-examples.md · references/
• test-play-model.md · references/
• arena.sh · scripts/

SKILL.md (first lines)

# Skill Arena
> Test play for skills and deck configurations. Not "which is best" — "which is best for what."

> **Invocation.** `arena <cmd>` below is shorthand for
> `bunx @lythos/skill-arena@0.21.0 <cmd>` (no install), or
> `lythoskill-arena <cmd>` if installed globally. The bare `arena` name is NOT
> a registered bin — it is an in-repo alias convention in the lythoskill
> monorepo. `deck <cmd>` likewise → `bunx @lythos/skill-deck@0.21.0 <cmd>`.
> Want the shorthand? `alias arena='bunx @lythos/skill-arena@latest'` — optional.

## Decision Tree (READ FIRST)

```
User says: "test/compare/arena/benchmark/A vs B"
    │
    ├── Cross-PLAYER? (kimi vs codex vs claude)
    │   OR user explicitly says useAgent/specific player
    │   OR platform doesn't support Agent tool subagents
    │     → CLI runner REQUIRED (useAgent → Bun.spawn)
    │     → bunx @lythos/skill-arena vs --config arena.toml
    │     → Each side spawns its player CLI process
    │
    └── Same player, different DECKS? (DEFAULT)
          → Agent-orchestrated — NO CLI
          → YOU spawn subagents via Agent tool
          → CLI prepare-workdir + CLI archive + parallel dispatch
          → Judge subagent collects + scores
```

## Default: Agent-Orchestrated (single & cross-deck vs)

**This is how arena works 95% of the time.** The agent and CLI operate as a two-way control transfer protocol. Agent delegates mechanical invariants to CLI. CLI hands control back via its exit paths (success → next step; error → fix command). Agent stays in its own main loop — the subagent pattern is container spawn, not external RPC.

```mermaid
flowchart TD
    A["🤖 Agent: parse request"] --> B{Cross-PLAYER?}
    B -->|Yes| C[🔧 CLI vs --config]
    B -->|No — DEFAULT| D["🤖 Agent → 🔧 CLI: prepare-workdir"]
    D -->|"✅ workdir ready"| E["🤖 Agent: spawn subagents"]
    E --> F["🤖 Subagents: execute + write artifacts"]
    F --> G["🤖 Agent: collect + spawn judge"]
    G --> H["🤖 Judge: score → report.md"]
    H --> I["🤖 Agent → 🔧 CLI: archive"]
    I -->|"✅ archived"| J["🤖 Agent → 🔧 CLI: deck link restore"]
```

**The protocol in one line**: Agent hands to CLI (`prepare-workdir`, `archive`, `deck link`). CLI exits with success (`✅ workdir ready → spawn`) or HATEOAS error (`❌ missing --deck → here's the fix command → retry`). Agent reads the exit, decides next move, continues. Three CLI exit points, three handoffs back to agent.

### single — test one deck

```
🤖→🔧 prepare-workdir --out /tmp/arena-xxx --brief "task"
    CLI exits: ✅ Workdir ready → 🤖 spawn subagent
🤖 Agent tool spawn: subagent executes in workdir, writes artifacts + decision-log-&lt;cell-id&gt;.jsonl
🤖→🔧 archive --from /tmp/arena-xxx --to ./playground --sides side-a
    CLI exits: ✅ Archive complete → 🤖 done
🤖→🔧 deck link parent deck (restore)
```

### cross-deck vs — compare decks A vs B

```
🤖→🔧 prepare-workdir × N (each side isolated, each with own deck)
    CLI exits: ✅ Workdir ready × N → 🤖 spawn N subagents in parallel
🤖 Agent tool spawn ×N, run_in_background=true
🤖 Collect: merge each side's per-cell decision logs → {side}/decision-log.jsonl
🤖 Spawn judge subagent: score per criteria → report.md
🤖→🔧 archive --from /tmp/arena-xxx --to ./playground --sides side-a,side-b
🤖→🔧 deck link parent deck (restore)
```

**Why agent-orchestrated is default**: Subagent = container spawn, not external RPC. Agent stays in its own main loop — can read subagent output, fix failures mid-run (switch mirror, adjust timeout, retry), spawn judge. Decision-log.jsonl from each subagent provides full observability. Cross-deck vs IS map-reduce — same agent type, different decks, parallel spawn, judge reduce.

## Cross-Player Mode (OPT-IN, CLI only)

Use ONLY when comparing different players (kimi vs codex vs deepseek vs claude). The Agent tool can only spawn the same agent type — it CANNOT simulate another CLI's memory, hooks, or tool-use semantics. This is a hard runtime boundary, not a preference.

```bash
# Single deck, explicit player
bunx @lythos/skill-arena@0.21.0 single \
  --deck ./skill-deck.toml \
  --brief "Investigate this repo" \
  --player kimi

# vs mode with arena.toml (each side's player in config)
bunx @lythos/skill-arena@0.21.0 vs --config ./arena.toml
```

See `references/player-setup.md` for player discovery, installation, and API key setup.

## Standard Posture: Arena as Mindset Validator

**Purpose**: Verify that a skill's **mental model** (SOP, behavior pattern, decision chain) actually shapes agent behavior — not just that the skill file is read.

**Minimal deck principle**: Include ONLY the governance skill (`lythoskill-deck`) and the target skill under test. Extra skills dilute the signal — you are testing whether the target skill's intent survives when no other skills are there to compensate.

**Standard posture** (4 steps):

1. **Prepare** — `prepare-workdir` with minimal deck
   ```bash
   bunx @lythos/skill-arena@0.21.0 prepare-workdir \
     --deck ./test-deck.toml \
     --out /tmp/arena-$(date +%Y%m%d-%H%M%S) \
     --brief "Execute the target skill's core workflow"
   ```

2. **Dispatch** — spawn subagent with decision-log mandate
   - Prompt MUST include: "Your working directory is {workDir}. Deck: {deckPath}. Task: {brief}. MANDATORY: write decision-log-{cellId}.jsonl to your CWD. Each line records a decision you made and why." — one cell id per cell, never a shared name.

3. **Observe** — collect decision-logs, not just artifacts
   - Read every `decision-log-*.jsonl` from workdir, or the merged `decision-log.jsonl` after the collect step
   - Check: did the subagent follow the skill's declared SOP?
   - Check: did the subagent stop at decision points and ask, or did it guess?
   - Check: are the decisions traceable to the skill's instructions?

4. **Judge** — score mindset alignment, not output correctness
   - "Did the subagent behave as the skill intended?" > "Was the output correct?"
   - A correct output achieved by guessing is a FAIL — the skill's mental model did not transfer.
   - A wrong output achieved by faithfully following the skill's SOP is valuable feedback — the skill's instructions need refinement.

From lythos-labs/lythoskill, MIT.

More from lythos-labs/lythoskill