llama-tune

Files (7)

• SKILL.md
• llama-cpp.md · references/
• llama-swap.md · references/
• _common.sh · scripts/
• bench.sh · scripts/
• model_info.py · scripts/
• verify_load.sh · scripts/

SKILL.md (first lines)

# llama-tune

Tune one model's config the way it's actually done: measure, don't guess. Read
the model's architecture, find its recommended settings, then change ONE dial at
a time and benchmark, confirm the winners work together, and verify a real load
fits with headroom before writing the config.

## Before anything: don't contend for the GPU

Benchmarks are meaningless if something else is using the cards. **Confirm the
rig is idle before any bench or load** (`llama-server --list-devices` should show
cards near-fully free; ask the user if unsure). CPU-only steps (reading
metadata, editing config) are always safe.

## Pick the mode

Ask the user which, if not stated:

- **bench** — maximize tok/s for leaderboard/benchmark runs. Smallest card
  footprint that fits, context only as large as needed, KV quant chosen purely
  for speed. Squeeze.
- **agentic** — maximize usable context for a coding agent, kept reliable.
  Largest context the model (native cap) and hardware allow, **prompt cache ON**,
  KV quant that preserves quality, and a VRAM headroom floor so it won't OOM mid
  run.

## Workflow

### 1. Resolve the model + read its architecture (CPU only)

Find the GGUF path (from the llama-swap config key, or ask). Then:

```
scripts/model_info.py <model.gguf> --vram-mib <per-card-total> --cards <N>
```

Get `--vram-mib` (per-card TOTAL) and `--cards` from `llama-server
--list-devices`. This prints arch, weights size, layers, KV heads, native
context, MoE/SSM/SWA flags, KV/token, **min cards needed**, and rough
max-context floors per (cards, headroom, KV-quant).

Read `references/llama-cpp.md` "Architecture gotchas" — MoE, hybrid/SSM, and SWA
make the KV estimate a loose upper bound; real footprint is usually much smaller,
so the only trustworthy fit number comes from step 5.

### 2. Look up the Hugging Face model card

Search/fetch `https://huggingface.co/<org>/a model` (and the GGUF repo, e.g.
unsloth/bartowski) for **recommended inference settings**: temperature, top-p,
top-k, min-p, repeat-penalty, chat template / `--chat-template-kwargs`,
reasoning format, and any "must use these" notes. Use WebFetch. These set the
sampling flags; the skill tunes performance, not quality — keep the card's
sampling unless the user says otherwise.

### 3. Detect hardware + set the target

From `--list-devices`: card count and per-card total/free VRAM (don't assume —
rigs differ). For **agentic** mode, ask the user **squeeze vs conservative**
headroom and set a floor (e.g. ~1 GiB squeeze, ~2 GiB balanced, ~3 GiB
conservative) that must hold on the *tightest* card (the main device runs
tighter — see references/llama-cpp.md).

### 4. Bench each dial — ONE at a time

Establish a baseline (current config), then change a single dial and re-bench.
Use `scripts/bench.sh a model <devices> <ts> [flags]` (prints clean pp/tg rows).
Dials, in rough priority:

1. **Card count** — fewest cards the weights fit on (min from step 1). For
   batch=1, fewer cards = faster (pipeline overhead). Bench 3-card vs 2-card vs
   1-card as applicable.
2. **KV quant** — `f16` vs `q8_0` (`-ctk/-ctv`). f16 is often equal/faster and
   higher quality when it fits; q8_0 when context needs the space.
3. **ubatch** — `-ub 512,1024,2048` for prefill (llama-bench accepts a list).
4. **flash-attn**, **-sm row** — only if a model misbehaves or to squeeze more.

Record what helps / hurts / is neutral. Keep gains that are real (outside the
± noise) and free.

### 5. Combine winners + verify a real load (ground truth)

Benchmarks don't prove a full-context config loads and survives prefill. Take
the winning dials together and run:

```
scripts/verify_load.sh a model <devices> -ts ... -np 1 -ctk ... -c <ctx> ...
```

It reports **MIN free VRAM across cards**, runs a generation, and checks
prompt-cache reuse. Require:
- loads + generates coherently,
- MIN free ≥ your headroom floor (agentic) — if not, drop context or add a card,
- for agentic: `req2 cached_tokens > 0` (prompt cache reuse working). If a config
  has `--no-cache-prompt`, test removing it (see references/llama-cpp.md).

For **bench** mode, also bench at the workload the leaderboard uses (matching
prompt/output token counts) so the reported numbers line up.

### 6. Apply + re-verify

Edit only the target model's `cmd:` block in the llama-swap config (see
`references/llama-swap.md` for structure and house rules), add `-np 1`, set the
tuned flags, and write a comment summarizing: card count + why, KV choice,
context, before→after tok/s. Then run `verify_load.sh` once more on the
final block to confirm. Tell the user to restart llama-swap.

## Mode decision cheatsheet

| dial | bench (max tps) | agentic (max ctx, reliable) |
|---|---|---|
| card count | fewest that fit (fastest) | fewest that fit *with headroom at full ctx* |
| context | only as large as the bench needs | model native cap, as high as headroom allows |
| KV quant | whichever decodes faster | f16 if it fits, else q8_0 (preserve quality) |
| `-np` | 1 | 1 |
| prompt cache | off is fine (single-shot) | **on** (`--cache-reuse`, no `--no-cache-prompt`) |
| headroom | minimal | enforce the floor on the tightest card |

## Scripts

From 1337hero/claude-toolkit, WTFPL.

More from 1337hero/claude-toolkit