functional-area-resolver

Files (2)

• SKILL.md
• routing-eval.jsonl

SKILL.md (first lines)

# Functional-Area Resolver — Pattern for Compressing Routing Tables

## Problem

Routing files (RESOLVER.md, AGENTS.md) grow as skills are added. Each skill
gets its own row (trigger -> skill path). At ~200+ skills this hits 25-30KB,
eating context budget that should go to actual work.

## Solution: Functional-Area Dispatchers

Replace N rows per area with **one entry per functional area**. Each entry
lists all sub-skills it can dispatch to in a `(dispatcher for: ...)` clause.

### Before (270 rows, 25KB)
```
- Creating/enriching a person or company page -> `enrich`
- Fix broken citations in brain pages -> `citation-fixer`
- Publish/share a brain page as link -> `brain-publish`
- Generate PDF from brain page -> `brain-pdf`
- Read a book through lens of a problem -> `strategic-reading`
- Personalized book analysis -> `book-mirror`
- Brain integrity -> `brain-librarian`
...
```

### After (13 rows, 13KB)
```
- **Brain & knowledge**: create/enrich/search/export brain pages, filing,
  citations, publishing, book analysis, strategic reading, concept synthesis,
  archive mining -> `brain-ops` (dispatcher for: enrich, query, brain-pdf,
  brain-publish, brain-export, brain-librarian, citation-fixer, book-mirror,
  strategic-reading, concept-synthesis, archive-crawler, ...)
```

## Why It Works

The LLM doesn't need one row per sub-skill. It needs:
1. **Area recognition** — "this is about brain pages" -> Brain & Knowledge
2. **Sub-skill visibility** — the `(dispatcher for: ...)` list shows what's available
3. **The skill file itself** — once the LLM reads `brain-ops/SKILL.md`, it has full routing detail

This is a **two-layer dispatch**: routing file routes to the area, the area
skill routes to the specific sub-skill. Each layer does one job well.

## A/B Eval Results

Three resolver architectures tested across three Anthropic frontier models
(Opus 4.7, Sonnet 4.6, Haiku 4.5) on real production AGENTS.md content,
20 hand-authored training fixtures + 5 held-out blind fixtures, n=3 seeded
repeats per (fixture, variant). Two scoring rules: **STRICT** (predicted
slug exactly equals expected) and **LENIENT** (predicted is in the same
dispatcher area as expected). Both matter:

- STRICT measures: "does the LLM return the exact slug?"
- LENIENT measures: "does the LLM land in the right area, even if it picks a
  more-specific sub-skill from `(dispatcher for: ...)`?" This is closer to
  production behavior — an agent that lands in `gmail` for an email intent
  succeeds even if the resolver entry said `executive-assistant`.

### Training corpus (n=20, 3 seeds × 3 variants × 3 models, LENIENT)

| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 | Size |
|---|---|---|---|---|
| baseline (270 bullet rows) | 81.7% ± 7.2% | 86.7% ± 7.2% | 73.3% ± 7.2% | 25KB |
| **functional-areas** (this pattern) | **98.3% ± 7.2%** | **100% ± 0%** | **88.3% ± 7.2%** | **13KB** |
| resolver-of-resolvers (no dispatcher clause) | 63.3% ± 14.3% | 41.7% ± 7.2% | 65.0% ± 12.4% | 10KB |

### Held-out blind corpus (n=5, 3 seeds, LENIENT)

| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 |
|---|---|---|---|
| baseline | 100% ± 0% | 100% ± 0% | 100% ± 0% |
| **functional-areas** | **100% ± 0%** | **100% ± 0%** | **100% ± 0%** |
| resolver-of-resolvers | 100% ± 0% | **73.3% ± 28.7%** | 100% ± 0% |

### What the data shows

1. **Functional-areas BEATS baseline on training across all three models** (+13 to +17pp) at 48% the size. Held-out is saturated at 100% for both — within margin of error.

2. **The `(dispatcher for: ...)` clause is the load-bearing signal.** resolver-of-resolvers strips that clause and collapses to 41.7% on Sonnet — the catastrophic failure case the original PR predicted, now observed.

3. **The pattern works because the LLM can drill into the dispatcher list.** Most "STRICT failures" are the LLM picking a more-specific sub-skill (`gmail` instead of `executive-assistant`). That's the pattern working as designed. STRICT scoring under-counts; LENIENT scoring reflects production agent behavior.

4. **The pattern's value scales with model tier.** Compression gain (functional-areas vs baseline, training, LENIENT) is +17pp on Opus, +13pp on Sonnet, +15pp on Haiku. Sonnet shows the cleanest separation between functional-areas and resolver-of-resolvers (100% vs 41.7%) — model capacity affects how much the dispatcher signal matters.

### Reproduce

```bash
cd evals/functional-area-resolver
node harness.mjs --model opus    # ~225 LLM calls, ~$1.70 at Opus pricing
node harness.mjs --model sonnet  # ~$1.00
node harness.mjs --model haiku   # ~$0.30
node rescore.mjs baseline-runs/2026-05-11-opus-4-7.jsonl  # zero-cost re-score
```

Receipts (model, prompt_template_hash, fixtures_hash, harness_sha, ts):
`evals/functional-area-resolver/baseline-runs/2026-05-11-{opus-4-7,sonnet-4-6,haiku-4-5}.jsonl`.

### Methodology caveats

- **Production prompt matters.** With a naive "return the skill slug" prompt
  (no instruction about `(dispatcher for: ...)`), every compression variant
  collapses to ~30-60% on Opus. The dispatcher-aware prompt is in
  `evals/functional-area-resolver/harness-runner.ts:PROMPT_TEMPLATE`. Use it
  as the template for your agent's harness; without it, compression breaks.
- **Training corpus and variants were authored by the same release.** Held-out
  corpus was written before the variants and never adjusted; this mitigates
  but does not eliminate overfitting.
- **Confidence intervals via t-distribution across n=3 seeded repeats.** Hold the
  n=3 lower-bound: high CIs mean the underlying sample is noisy.
- **Single-vendor result.** All three models are Anthropic. Cross-vendor
  verification (Gemini, GPT) is a v0.33.x follow-up.
- **Held-out blind set is small (n=5).** Saturated at 100% across most cells —
  the harness can't distinguish between "100%" and "95% with one nondeterministic
  miss." Expanding to ≥20 is a v0.33.x follow-up.

### Prior work and citations

The pattern is a **static-prompt analog of hierarchical agent routing**, a

From Ninatuzi/gbrain, MIT.