lemonade-router-builder

Files (7)

• .federated.json
• SKILL.md
• evals.json · evals/
• examples.md
• reference.md
• validate.py · scripts/
• skill-card.md

SKILL.md (first lines)

# Lemonade Router Config Generator

Generate a **`collection.router` policy JSON** from a plain-English description
of how requests should be routed. The skill produces and validates the JSON
only - it does not call the live server, register the policy, or run requests
through it. The JSON is accepted by the strict server-side parser on the first
try and stays editable in the desktop app's Hybrid Router editor.

## Prerequisites

- **Lemonade Server v11.5.0+** running locally (`lemonade server start`).
  Required only to register and test the generated policy - the skill itself
  (JSON generation + offline validation) works without a live server.
- **No GPU or ROCm dependency** for authoring. The router policy is a JSON
  document; no hardware is needed to generate or validate it.
- **Python** (any 3.x) in PATH - used by the bundled offline validator
  (`scripts/validate.py`). No extra packages required.

The router picks one **candidate** model per request. Two authoring modes
exist, and choosing the right one is the first decision:

| Mode | JSON shape | When |
|------|-----------|------|
| **LLM-as-router** | `routing.router` block | The user describes intent only by *meaning* ("sensitive", "hard questions", "creative writing") with no concrete signals. A small LLM reads each prompt and picks the candidate. |
| **Rules** | `routing.rules` (+ optional `routing.classifiers`) | The user names any concrete signal: keywords, regex, length, tools, images, metadata, PII/topic classifiers, thresholds, "first match", fallback logic. Deterministic, no extra LLM call for simple conditions. |

`routing.router` is **mutually exclusive** with `routing.rules` and
`routing.classifiers` - never emit both.

## Step 1 - Extract from the user's words

- **Candidates**: the models that may *answer* requests. Verbatim model names
  (e.g. `Gemma-3-4b-it-GGUF`). If the user names none, ask - never invent
  model names. `lemonade list` or `GET /api/v1/models` shows what's available.
  A name the user *did* give may still not exist on the target host - the
  offline validator can't check that (Step 8b closes the gap).
- **Default / fallback**: which candidate gets everything that matches nothing.
  If unstated, use the model the user framed as "local", "small", or "safe";
  otherwise the first candidate mentioned.
- **Signals**: every condition mentioned (keywords, patterns, length, images,
  tools, topics, PII, safety) and which model each one routes to.
- **Classifier models**: models named for *detection* rather than answering
  (BERT-style encoders, embedding models, an LLM used as judge).

## Step 2 - Scaffold

Always exactly this envelope (the parser rejects unknown or missing keys):

```json
{
  "version": "1",
  "model_name": "user.MyHybridRouter",
  "recipe": "collection.router",
  "components": [],
  "routing": { }
}
```

- `version` is the literal string `"1"`.
- `model_name` must start with `user.`; slug from the user's description if
  they gave a name (`user.<Name>` using only `[A-Za-z0-9._-]`). If they didn't
  name it, derive one from context instead of a fixed literal - e.g.
  `user.<slug-of-default-candidate>-Router` - so two different policies don't
  collide by default. **`/pull` is idempotent per `model_name`: registering a
  second policy under the same name silently overwrites the first.** If this
  conversation already produced an unnamed router, don't reuse the same
  derived name for the next one - ask, or pick a visibly different name.

## Step 3 - Candidates and default

```json
"candidates": ["<answering models>"],
"default_model": "<one of candidates>"
```

`default_model` MUST be listed in `candidates`. Candidates should be
chat-capable LLMs - not embedding, classification, or image models.

## Step 4 - Mode A: LLM-as-router

```json
"router": {
  "type": "llm",
  "model": "<small chat LLM>",
  "prompt": "You route user requests to the best model. <one sentence per candidate: when to pick it, using the exact model name>."
}
```

- `model` defaults to the **most capable candidate**, not the cheapest one.
  The router judges every single request that flows through the policy, so a
  weak judge silently misrouting everything is a worse default than the extra
  cost of a stronger one. State the choice in the summary you give the user -
  `router.model: <chosen>` (most capable candidate available; pick a smaller
  dedicated judge model yourself for lower per-request cost, at the risk of
  the failure mode below). If the user already named a separate model for
  this role, use that instead of a candidate.
- If `default_used` stays `true` across varied test prompts even after fixing
  the prompt (see the bullet below and Step 9), the fix is a more capable
  `router.model`, not a further prompt edit - this is a judge-model-capability
  limit, not something prompt wording alone can solve.
- **Write intent only - never specify a reply format and never use imperative
  "Pick X" phrasing.** The engine unconditionally appends its own contract
  after your prompt: it lists the candidate names and demands a strict JSON
  reply `{"model": "<name>", "rationale": "<one sentence>"}`, then falls back
  to `default_model` on any deviation. A prompt that says "reply with ONLY the
  model name", "Pick Model-A", "respond with the model name", or similar is
  wrong about the wire format and causes weaker judge models to reply with a
  bare string that fails to parse - silently falling back to `default_model`
  on every request with no visible error.

  **Bad** (do not write): `"Pick Qwen3.5-9B-GGUF for sensitive queries, pick Qwen3.5-9B-NoThinking for everything else."`
  **Good**: `"Route to Qwen3.5-9B-GGUF when the request appears sensitive or contains personal information. Route to Qwen3.5-9B-NoThinking for all other requests."`

  Only describe *when* each candidate is appropriate. Never say "pick", "output", "reply with", or "respond with".
- **NEVER emit `rules` or `classifiers` in this mode.** The `routing` object
  in Mode A must contain exactly: `candidates`, `default_model`, and `router`.
  Adding `rules` or `classifiers` alongside `router` is a schema violation
  that the server parser rejects. If you catch yourself writing both, stop and
  remove `rules`/`classifiers` entirely.

From amd/skills, MIT.

More from amd/skills