optimize

Files (1)

• SKILL.md

SKILL.md (first lines)

# optimize

The improvement loop is code (`hbs optimize`), not opinion. Your job is to read it, explain it in plain words, and bring the human the decisions that are theirs. Never edit `tuning.yaml` by hand and never accept an experiment yourself.

## Read

- `hbs optimize` — scorecard (last 30 days vs the 30 before), running experiments, accepted overrides, proposals.
- `hbs optimize --json` — the same, machine-readable.
- `hbs report --repo <repo>` — one project's per-phase numbers.

## The four goals and what moves them

| Goal | KPIs | Levers the system pulls |
|---|---|---|
| **Cheaper** | tokens and reported $ per shipped idea; cache hit ratio | one-step-cheaper model on nodes that already succeed (auto A/B); quick tier for features; incremental skip of fresh nodes |
| **Faster** | idea lead time; card run time; queue wait; your gate wait | parallelism (`kanban.max_in_progress`); quick tier; lane switches for slow nodes |
| **More accurate** | verify first-pass; review rounds; retries; gate rejects; skill eval scores | stronger model / different-vendor lane on struggling nodes (you approve); lessons; eval-gated skill evolution |
| **Hallucinate less** | dead or invented links + wrong license claims; claims without a source; "done" claims that verify failed | `hbs audit` gate on research cards; cross-vendor verify; tamper canary in evals; lanes with bad audit rates get re-routed |

Accuracy is a constraint, never a trade: an experiment that saves tokens but loses >10 points of success, or adds bad links, **loses**. Accepted changes stay armed for auto-rollback.

## What to bring the human

1. Anything that got **worse** on the scorecard, with the likely cause (a new node, a new lane, a model change in `history`).
2. `EXPERIMENT WON` lines → "Accept? `hbs optimize accept <id>`" with the numbers.
3. Proposals marked "need you" → one line each: what, why, the command.
4. `ROLLED BACK` lines → what was undone and why; nothing for them to do.

## Levers only the human pulls

- Adding cheaper models to `ladders` in `~/.build-framework/tuning.yaml` (only ids that exist in their Hermes providers).
- `settings.auto_start` (default on for cheaper/faster), `share`, `min_samples`.
- Choosing the tier per idea (`hbs go --tier quick|full`).

## Honest limits to say out loud

- Verdicts need ~6 samples per arm; with few runs the scorecard shows "—" and that is correct.
- On subscriptions (Codex/ChatGPT, Claude Max, SuperGrok) the marginal $ is ~0 and cost shows as tokens; fewer tokens = more runs before rate limits.
- "Hallucination" is measured by proxies (dead links, license mismatches, false "done" claims), not every false sentence.

From Kshintak/new-build-framework, MIT.

More from Kshintak/new-build-framework