optimizing-model-routing

Files (17)

• SKILL.md
• evals.json · evals/
• expected-eval1.json · evals/
• expected-eval3.json · evals/
• AGENTS.md · evals/fixtures/miniframework/
• build.md · evals/fixtures/miniframework/agents/
• council-security.md · evals/fixtures/miniframework/agents/
• council-ux.md · evals/fixtures/miniframework/agents/
• council.md · evals/fixtures/miniframework/agents/
• design.md · evals/fixtures/miniframework/agents/
• triage.md · evals/fixtures/miniframework/agents/
• benchmarks.md · evals/fixtures/miniframework/reference/catalog-snapshot/
• go-catalog.json · evals/fixtures/miniframework/reference/catalog-snapshot/
• zen-catalog.json · evals/fixtures/miniframework/reference/catalog-snapshot/
• model-routing.md · evals/fixtures/miniframework/reference/
• benchmarks.md · references/
• verify_rebind.py · scripts/

SKILL.md (first lines)

# Optimizing Model Routing

Re-derive the library's role → model bindings from objective evidence: the
opencode catalogs plus the most objective benchmark data available.
**Coverage is skill-level** — the approval table lists every skill against
its role, so nothing hides behind indirection. **Mutation is role-level** —
updates land on `reference/model-routing.md` role rows and `agents/*.md`
model lines, keeping the roles-not-model-IDs single source of truth. The
pass stops at user approval; no binding is edited before it.

## The routing pass

Copy this checklist and check off items as you complete them.

```
Routing Progress:
- [ ] 1. Scope the pass (all roles, or one named role)
- [ ] 2. Read current state
- [ ] 3. Gather catalogs + liveness (live fetch; docs + routing probe)
- [ ] 4. Identify + verify objective benchmarks
- [ ] 5. Rank candidates per role (library default first — Go, with free as the opt-in column; Zen/direct escalation — live candidates only)
- [ ] 6. Capture the evidence (table + Questionable/Uncertain; write no files)
- [ ] 7. Present table + evidence + Questionable/Uncertain; STOP for approval
- [ ] 8. On approval: apply, verify, commit, PR
```

### Step 1 — Scope the pass

Default: every role in `reference/model-routing.md`. A named role (or a
skill named by the user) scopes the pass to that role only — analyze it,
present it, touch nothing else, and say so explicitly so a scoped pass is
never mistaken for a clean bill of health on the rest.

### Step 2 — Read current state

Read `reference/model-routing.md` in full — the bindings table and **hard
exclusions** — and every `model:` line in `agents/*.md`. Two absolute rules from this step:

- **Hard exclusions win over any score.** An excluded family is dropped
  from candidacy regardless of benchmark results, even the best ever
  reported.
- **A bound ID missing from the catalog OR failing liveness is a stale binding.** A stale binding is any ID that is not catalog-listed, not docs-listed (free tier), or fails to route in a live opencode session (see Step 3). It can only be flagged and rebound, never re-proposed.

### Step 3 — Gather catalogs + liveness

Production default: fetch both live catalogs **and** the docs free list
(`https://opencode.ai/docs/zen/#endpoints` — authoritative for free-tier;
see `reference/model-routing.md` Update procedure). Cross-check catalog
vs docs; any divergence is a catalog-vs-availability mismatch → record in
Questionable/Uncertain and exclude the ID from candidacy. Then
liveness-probe every remaining candidate (minimal routing test in a live
opencode session: select an agent bound to that model / send a trivial
prompt and verify clean routing — no error and no opt-in, consent, or
region gate). **Only live-verified IDs are
candidates** — catalog membership alone never makes a model a candidate.
Access restrictions fail the gate: a model routable only behind an
explicit opt-in, consent gate, or region lock is **not live** — never opt
in on the user's behalf; exclude the ID, name the gate in
Questionable/Uncertain. Gates lift without notice: every pass re-probes
from scratch, and a rehabilitated model re-enters candidacy automatically.

**Go/Zen escalation gate:** for any model proposed for a Go or Zen
escalation row, run a live routing probe on its tier (select an agent
bound to that model on Go/Zen, invoke with a trivial prompt, confirm no
routing error). Record the probe date + result in the routing update
note. Non-live escalation IDs are excluded from the row.

Offline / snapshot mode: a local snapshot directory containing
`zen-catalog.json`, `go-catalog.json`, and `benchmarks.md` — use it when
the user points at one or fetching fails. State which mode ran; never
silently mix snapshot and live data. Every candidate must be
catalog-verified **and** live-verified before ranking; record the
retrieval date for the presentation and the routing update note.

### Step 4 — Identify + verify objective benchmarks

Read [references/benchmarks.md](references/benchmarks.md) now — the
objectivity hierarchy and the role → dominant-trait map. Core discipline:
every cited score carries source + date + independence status, and
**vendor self-reported numbers never decide a binding** — they are a tier
filter and belong in Questionable/Uncertain. Search for the most
objective evidence available before settling for less; if the best
evidence for a candidate is vendor-only, say so instead of laundering it
into a decision.

### Step 5 — Rank candidates per role

Per role in scope, rank **only over live candidates** (Step 3) on the
role's dominant trait per the reference map, deciding on the most
objective evidence available and tie-breaking upward in objectivity (an
independent result beats a cross-checked one beats a vendor claim).
Propose the best live model for the **default** column first, per the
library's recorded policy (`reference/model-routing.md`: Go default, free
as an explicit opt-in); a free model is proposed for the opt-in column
only, and Zen/direct rows are escalation picks. **Liveness is a hard
gate, like capability** — a non-live ID is not a candidate at any
benchmark score and never enters ranking. Capability requirements (e.g.
native image input for vision seats) are hard gates, not scores — a
text-only model cannot hold a vision seat at any benchmark number.
**Hard exclusions sit above all scores** — an excluded ID is dropped from
candidacy regardless of a live probe or a top benchmark result.

### Step 6 — Capture the evidence

Assemble the pass record as the Step 7 presentation itself — table,
evidence summary, and Questionable/Uncertain, each entry carrying
source + date + independence status. On approval, the *decision* is
recorded in the commit message and PR description (Step 8). Write no
files before approval.

### Step 7 — Present; STOP

Present three things:

- a simple table: role → used-by skills → current → proposed
  (free / Go / Zen) → bench basis (the one decisive number + its source);
- a brief evidence summary — one line per decisive benchmark, with source
  and independence status;
- a **Questionable / Uncertain** section — mandatory, never omitted:

From hierovision/ai-framework, MIT.

More from hierovision/ai-framework