AGENTS.md · jaredpalmer/kev

Sections

1. Commands
2. Skills
3. Layout
4. Notes
5. Calibration Research
6. Writing

First lines (shown under the repo's Apache-2.0 license)

# Kev — small Jev-style decision models

Causal LM + LoRA run prefill-only with a block-causal mask (shared state prefix, isolated question branches) and a
pointer readout over option boundary tokens, trained with log loss on public datasets plus generated policy and rule
records. No text generation. The released family is Qwen3.5 (0.8B / 4B / 9B) plus Kev-27B on the post-trained Qwen3.8-27B; `kev-0.5b` (Qwen2.5-0.5B, the prototype
this started as) and the Qwen3 generation are superseded but still published.
See README.md (deep dive), PLAN.md (where the research stands, what we have learned, the standing rules and what is next;
the full dated record of rounds 4-18 and everything before is frozen at git tag `research-archive-2026-09-24`),
docs/autoresearch.md (the operating program for an unattended research session) and docs/model-cards/ (one card per
checkpoint: recipe + metrics). README follows the Vercel Labs house style (tagline, for-the-badge badges, Highlights,
Title Case sections, API tables, Authors + License); model cards are formal.

## Commands
- Env: `uv sync` (add `--extra serve` for FastAPI + the TypeSafe SDK; on Apple Silicon it also pulls `mlx-lm` for the MLX backend, `--extra mlx` alone for library use). `kev` is a real package (setuptools, installed
  editable by `uv sync` since #14), Python 3.12 or 3.13 (`requires-python = ">=3.12,<3.14"`; 3.14 has no wheels for the pinned torch, `.python-version` selects 3.13,
  `uv sync --extra serve --python 3.13` overrides explicitly), `transformers>=5.17,<6`, `peft>=0.21`, `torch>=2.6,<2.9`. The dev
  group carries pytest, matplotlib and `modal==1.5.5`.
- Train: `uv run python -m kev.train --suite evals/v7/decision-v7 --base Qwen/Qwen3.5-4B-Base --base_revision <sha> --epochs 2
  --lr 5e-5 --batch 4 --accum 2 --dtype bf16 --checkpointing 1 --p_none_pair 0.25 --device cuda --out runs/kev-4b`
  (the released 4B/9B recipe; 0.8B uses `--lr 1e-4 --batch 8`). Records come from `--suite` (its training partition),
  `--data` (your own JSONL, optionally `+ --suite --replay N`), or built on the fly from the public sources
  (`--n_per_source`, the smoke path). `--help` lists everything; the knobs that matter:
  - losses: `--perm_kl/--perm_frac` (permutation KL), `--ord_w` (ranked probability score for Score),
    `--anchor/--anchor_w/--anchor_sources` (KL toward the frozen base's zero-shot answers, from `kev.anchors`),
    `--label_smoothing/--brier_w/--focal_gamma` (the round-3 calibration screen; all default 0 and none is in a release).
  - augmentation / mix: `--p_none`, `--p_none_distract`, `--p_distract`, `--p_none_pair`, `--synthetic_repeat`,
    `--public_frac`, `--train_sources`, `--holdout`. `--none_pair_max_state N` (plan key `none_pair_max_state`, needs
    `--p_none_pair > 0`; absent = every record may pair, today's recipes byte for byte): only records whose state has at
    most N tokens (encode's count, `<state>` included) emit none pairs, drawn from each record's own stream
    (`kev.train.none_pairs`), and `--length_sort` counts their two siblings (the state twice more) in the record's cost, so a
    pass cannot outgrow the cost its run was cut to. A pair repeats the whole state, so on long states it tripled a pass
    past the GPU (round 21's projection). `--pass_tokens_max N` (plan key `pass_tokens_max`, needs `--length_sort 1` and a hybrid
    base, whose passes are always rows or a shared prefix, what `pass_tokens` measures (an attention-only base runs the packed
    mask, refused once the model is built); not with `--row_budget` or `--perm_kl`; absent = today's plan byte for byte): the plan cuts each step on exact token shapes
    (`kev.train.plan_shapes`: every variant the epoch trains, siblings included, branches encoded without their state, the
    state counted once by `state_token_counts`) instead of characters, and a step whose costliest pass is over N padded
    tokens (`pass_tokens`) gets one more micro-batch per rank until none is (every rank the same count; a record over N
    alone is refused before the first step). Round 21 ran out of memory at its 62nd step on a characters plan: a
    token-dense PII state set the padding of 16 states at 5,877 tokens, a 98.7k-token pass costed in characters like its
    slot's other passes of 33-50k tokens.
    27B, 8 H200: N = 40,960 peaks at 101.6-108.2 GiB per GPU on the worst passes the plan allows
    (`scripts/sft_probe.py --passes`, `runs/sft-probe/r22-ceiling-27b-8xh200`).
  - architecture / precision: `--lora`, `--lora_targets all|dense|attn|qv` (`dense` freezes the DeltaNet projections on
    hybrid bases), `--head_dim`, `--option_isolation`, `--special_embeddings`, `--dtype` (autocast) vs `--weights_dtype`
    (frozen backbone; bf16 is required by the fused MoE experts of 35B-A3B).
  - full-weight: `--full_ft 1 --weights_dtype bf16` trains the whole text backbone (vision tower, LM head and MTP are never
    loaded) plus the head, with `kev.full_ft.MasterAdamW` (fp32 masters + moments; one GPU: in host memory, AutoJev's
    technique; under `torchrun`: FSDP2 shards, state on the GPUs, the head replicated). `--shared_prefix` (default on with
    `--full_ft 1`; hybrid backbones) runs each record's state once and its question branches from it (`kev.shared_prefix`:
    the DeltaNet recurrent and conv state and the attention keys/values of the state, functional, one checkpointed step per
    layer), exact to the row form (tests/test_unit.py, tests/test_model.py). `--length_sort 1` deals each step's records
    into micro-batches of neighbours in length balanced by padded cost (`--batch` becomes the average), so no rank waits
    long on another. `--row_budget N` (one GPU): padded tokens per forward/backward pass, a record that does not fit is
    split by question (27B on one H200 needs 8192; not with `--perm_kl` or `--anchor_w`). Needs torch >= 2.8. `--max_steps N` stops early. Resume: `--save_every_minutes M` /
    `--save_every_steps N` write `<out>/resume` (fp32 masters + moments per rank, scheduler, RNG, data position; under
    torchrun from a host copy in a background thread, the interval stretched so blocking stays under 5 %), `--resume 1`
    continues bit for bit (same arguments and world size), `--stop_after N` exits after a step. Snapshots:
    `--snapshot_fractions 0.25,0.5,0.75` (of the optimizer steps) / `--snapshot_every_steps N` write loadable bf16 checkpoints
    into `<snapshot_dir>/step-<N>/checkpoint` (N zero-padded to 7 digits like resume points, `step-0000389`; `--snapshot_dir`,
    default `<out>-snapshots`; `kev.full_ft.SnapshotWriter`; at most `kev.budget.MAX_SNAPSHOTS` = 8 per run, the disk they
    may take next to two resume points, refused at parse time or before the first step): the
    final checkpoint's files (save_pretrained shards + `head.pt` + tokenizer; `head.pt["snapshot"]` = step, steps, epoch
    fraction, records seen) plus `snapshot.json`, written last, which marks it complete. Under torchrun the ranks gather
    into rank 0's host memory (the only blocking part) and rank 0 writes in a background thread; one GPU writes before
    going on. Snapshots are never deleted; a continued run skips complete ones and rewrites an incomplete one, and a
    resume point's `latest.json` waits for the snapshot being written, so no committed resume point passes a missing
    snapshot. `training_metrics.json` has per-epoch gradient norms before clipping (`grad_norm`: mean, max, clipped
    steps), LoRA runs too; full-weight runs also `backbone_save_seconds` (the final gather + write) and `snapshots`
    (blocking and write seconds per snapshot of that attempt; each `snapshot.json` has its own). Trials: see Studies.
    27B, 8 H200, `--batch 8 --accum 2 --length_sort 1`, records shaped like the SFT corpus (`scripts/sft_probe.py --mix all`):
    7.6 records/s, one epoch of the corpus (192k records) ≈ 7.1 h / $293, 71 GB peak per GPU (`runs/sft-probe/sft2-*`,
    PR #125); one H200 (row form, PR #122) 0.84 records/s on ~1,000-token records.
    Long states (`scripts/sft_probe.py --state_tokens 16384,32768,49152,65536 --questions 3`, `--batch 1 --accum 1 --length_sort 1`, 8 H200,
    `runs/sft-probe/lc-27b-8xh200`): 1.12 / 2.33 / 3.79 / 5.45 s per record (14.8k-12.0k record tokens/s), 78 / 96 / 115 / 134 GiB peak per GPU
    of 140. A micro-batch whose states are unpadded runs the state through SDPA's causal flash kernel with no mask (`kev.shared_prefix`);
    with the explicit 64k x 64k mask a 64k step took 150 s instead of 44 and filled the GPU (`runs/sft-probe/lc-27b-8xh200-mask-ab`).
  - Only one training process at a time: two on MPS slow each other ~10x.
- Smoke: `uv run python -m kev.train --n_per_source 40 --accum 4 --out runs/smoke` (~1 min).
- Benchmark (the eval path for everything current): `uv run python -m kev.benchmark --run <run dir | Hub id[@rev]>
  --suite evals/v4/transfer-v4 --out runs/<name>`; `--remote <url>` scores any System One endpoint (`--remote-concurrency N` keeps N requests in flight; rows are identical), `--data x.jsonl` your
  own labelled rows, `--date_facts` the opt-in preprocessing, `--allow-test` is the only way to read a locked test.
  Writes `rows.json` (per question, with logits) + `report.json` (accuracy, ECE/Brier/NLL, selective coverage and AURC,
  permutation, isolation; `environment`: the kernel set the logits depend on, from `kev.predictors.kernel_environment`: package versions, GPU, backbone dtype, attention implementation, the DeltaNet layer forward (kev.fused_qwen35 replaces it) and the conv and delta rule transformers bound; compare it before pairing two reads bit for bit). `kev.calibrate --rows <rows.json>` reports what one temperature fitted on those rows would do (raw / shipped / workload in-sample / workload group-disjoint OOF, paired bootstrap vs shipped; report only, writes `calibration.json` next to the rows); external-suite `rows.json` are committed for this. `kev.evaluate` is the legacy prototype eval (`runs/kev`, `eval.json`) and is not used for
  releases. `kev.compare --candidate <dir> --reference <dir>` pairs two result dirs (record-clustered bootstrap).
  `kev.jev --suite ... --out ...` scores Jev through Vercel AI Gateway (AI SDK `experimental_evaluate`, node worker in
  `playground/scripts/jev-evaluate.mjs`; needs `AI_GATEWAY_API_KEY` or `--provision-scope`; budget-capped).
- Studies: `kev.experiment --plan experiments/*.json --suite <suite> --out runs/<study>` runs config-only trials
  (allowlist + ranges in `experiment.py: DEFAULTS/CHOICES/validated_trial`, provenance (`provenance.json`, copied into
  `result.json`; its `environment` is the kernel set the trial's reads were scored on, report.json's block from the same
  `kev.predictors.kernel_environment`, taken from the loaded predictor and replaced when a resume re-scores; trials before it have none), coverage/isolation gates,
  `results.jsonl` ledger, `--transfer <suite>` for an OOD read per trial, `--aggregate` to rank an existing directory,
  `--resume` for interrupted trials (evaluation; unfinished full-weight trials continue training from their resume point),
  `--wait-pid` to queue behind a training job). A full-weight trial runs under torchrun on every GPU of its container,
  writes a resume point every `experiment.RESUME_MINUTES` and is continued after a timeout by `kev.rounds watch`, not by
  Modal (`kev.budget`: `FULL_FT_RETRIES` continuations, up to 24 h per attempt, $1,000 per study; the bound counts every
  attempt): trials spawn with Modal's retries off, because Modal charged each killed timed-out attempt twice and gave
  round 22's trial two of its three attempts (`scripts/modal_retry_probe.py`); `runs/<study>.spawn.json` is the attempt
  ledger (`calls` = each trial's current call, `attempts` = all of them), and `modal_app.py::resume --trial <label>`
  (`modal_app.continue_full_trial`, what `watch` runs) spawns the next attempt only after a call ended by a timeout,
  within the ledger, with the study's GPU and timeout; `--beyond-bound` continues a trial without a ledger, uncounted. Each attempt is written to the ledger pending (a nonce) before its spawn; a pending entry blocks continuations until its call is adopted from the lease or it is `kev.budget.LEASE_STALE` old. The lease: every full-weight attempt holds `<study>/<trial>/attempt.json` on the `kev-leases` volume (`modal_app.TrialLease`: nonce, attempt, call id, heartbeat every `LEASE_HEARTBEAT` s from its own thread, marked ended on a clean exit); an attempt that finds another attempt's lease fresher than `LEASE_STALE` (900 s) returns `{"refused": ...}` without touching the trial, and `continue_full_trial` waits (bounded) for the old lease to end or go stale, so a continuation starts only after the old container is gone.
  It also keeps snapshots at `experiment.SNAPSHOT_FRACTIONS` (0.25, 0.5, 0.75 of its optimizer steps) in
  `<trial>/snapshots/step-<N>/checkpoint` (plan keys `snapshot_fractions` (a string, `"none"` for none) and
  `snapshot_every_steps`, which needs `max_steps` so `validated_trial` can check `MAX_SNAPSHOTS`; neither changes the
  config hash of a plan that omits it, nor the recipe). A 27B snapshot is ~51 GB of volume storage and never deleted: do
  not remove checkpoints or snapshots from the volume without asking. The volume copy is primary; a plan key
  `snapshot_hub_repo` (off by default) also mirrors each committed snapshot and the final checkpoint to that PRIVATE Hub
  model repo (`kev.mirror`, `modal_app.run_mirror`: a CPU container spawned after the commit, HF token from the Modal
  secret `huggingface-secret` or `KEV_HF_SECRET`; refuses a public repo, retries once, never fails training; the commit is
  recorded in `snapshot.json["hub"]` / `checkpoint/hub.json`, at `<study>/<trial>/<step-N | final>/` in the repo).
  `modal_app.py::mirror_snapshots --study X [--paths /runs/...checkpoint] [--repo jaredpalmer/kev-snapshots] [--dry-run]`
  (re)uploads existing ones (a 27B checkpoint is ~51 GB; ask before mirroring those).
  The container commits the runs volume after every completed resume point and snapshot (a timeout skips the final commit); an attempt
  that fails with an error writes `failed.json` and returns `{"failed": ...}` instead of raising, so it is never continued
  (`kev.rounds.poll_modal` reports it as a failure; a timeout it reports as `"timeout"`). `modal_app.py::pull` (and `kev.rounds watch`) leaves full-weight
  shards (`model*.safetensors` directly in a `checkpoint/` directory: final and snapshots) and resume points on the volume (`--weights` copies them);
  read a snapshot like any checkpoint: `::benchmarks --jobs "/runs/<study>/<trial>/snapshots/step-<N>/checkpoint@<suite>@<name>"`.
- Rounds (every registered experiment since round 5): one spec per round, `experiments/rounds/r<N>.json`, committed before any
  training or read: studies (plan, GPU, timeout, budget), arms (`<size>-<label>`: trial + parent), parents (trial + where each
  of its reads lives), read tags -> suites (`--allow-test` for test panels, `locked_test` for the locked read), the rule
  (panels of reads, bootstrapped metrics, criteria on paired bounds, rank) and confirmation stages. `kev/rounds.py` is the
  one engine: `uv run python -m kev.rounds {validate,launch,watch,launch-reads,readout,confirm} <spec>`; `watch` polls the
  spawned trials (state in `runs/<study>.watch.json`, resumable; DNS/connection errors retried, a trial's own exception is a
  failure, a timed-out full-weight trial gets its next attempt within the spawn record's ledger), pulls each finished trial's study (one pull per study at a time, `modal_app.pull_lock`) and launches its reads once (per-arm lock; the
  launch intent is written first to `runs/r<N>-reads-<arm>.json`, and an arm launched within its reads' timeout is not relaunched; a
  finished call that maps to no arm is logged and makes `watch` exit non-zero),
  60 s apart, then writes `runs/r<N>-readout/round<N>.json`; `confirm <spec> --stage <s>` writes `runs/r<N>-verdict/<size>-<s>.json`.
  Every side is served at the temperature fitted on its own development rows, unless the round registers a `temperature` pool
  for its arms (reads pooled, per-read `sources` allowlist, `exclude_reads`; refused inside any panel a temperature-dependent
  criterion reads; round 20). `validate` also refuses a pool that shares data with any arm's training (`kev.rounds.pool_conflicts`):
  (a) a pooled suite is an arm's training suite, a component its manifest names (sft-v1's `inputs.components`) or a plan's `data`
  suite; (b) it pools sources the arm trained on; (c) it reads the `calibration` or `development` partition of any training corpus
  (a suite with trainable sources). The arm's training comes from its study in the spec or its trial's `provenance.json` (manifest
  hash); a checkpoint arm is covered by the round's trial arms or names `trained_on`; training it cannot establish or list (no provenance,
  a manifest without sources, a `data` file outside `evals/`) is a problem for a new round, archived for a recorded one. Sources are
  compared by name, so pools use sources that are eval-only in Kev by construction; a `sources` allowlist must name sources its suite lists.
  Why: round 19 fitted its SFT arms' temperature on `sft-v1` development rows, held-out *items* of the training sources, so
  in distribution (T 0.955, breadth-v1 ECE 0.059); round 20's pool of held-out *datasets* gave 0.0085 on the same checkpoint.
  Each arm's read-out records `temperature_source` (the pool with its reads and question count, or its trial's development rows
  and their suite; parents in `parent_temperature_source`: their trial's development rows plus `shipped`, head.pt's T when
  it is local or in the HF cache) and the printed table warns (`!!!`) when an arm is served at a training corpus's rows. From
  round 21 (`POOL_REQUIRED_FROM`) a rule or confirmation criterion the temperature moves (anything but accuracy) without a
  `temperature` pool is a problem, so `validate` and `launch` refuse the round; rounds <= 20 only get the warning, so their
  specs keep validating. `validate`/`launch` also warn (report only) when a parent's served T, fitted on a training corpus's
  development rows, is more than 0.05 from its shipped head.pt T. A panel with
  `"by_length": true` (or `{edges, tokenizer}`) adds acc / ECE / Brier / confident errors per state-token bucket
  (`kev.metrics.calibration_by_length`: under_8k, 8k_16k, 16k_32k, 32k_64k, 64k_plus and the tails 8k_plus/16k_plus/32k_plus;
  tokens = the encoded state segment, `<state>` included, as `kev.model.encode` builds it and `kev.serve` reports it, counted from the reads' suite records with `kev.suite.ADMISSION_TOKENIZER` unless given), and a criterion may read a
  bucket (`long.ece_16k_plus.candidate <= 0.05`). Panel filters (round 24, `kev.rounds.panel_filter`; the same rows leave both sides):
  `source` (a name or a list), `exclude_sources`, `exclude_tasks`, and `exclude_file` `{path, sha256}` (a JSON of `ids` /
  `sources` / `tasks`, checked against its hash; a private list, e.g. tasksource-heldout families, stays out of git and is
  restored with `scripts/private_rows.py`, so the public spec carries only the path and a salted file's sha256); `"optional": true`
  marks a report-only panel that no criterion or rank may read and whose absent reads never leave an arm incomplete; the
  `temperature` pool's `ci: {level, samples, seed}` adds each arm's bootstrap interval of its pooled T (`temperature_ci`). An arm may be a checkpoint without a trial (`"checkpoint": "/runs/..."`, its "transfer" rows
  from a `transfer_read`), e.g. a WiSE-FT interpolation (`scripts/interpolate_checkpoint.py`, `modal_app.py::interpolate`: with the
  base, or `--toward` another checkpoint of the same base and revision, full weights or a LoRA merged in fp32, `--blend_head` to blend the
  pointer heads too; round 23); a LoRA checkpoint as a full-weight one, to start `kev.train --full_ft 1 --init_from` from it
  (`scripts/merge_lora_checkpoint.py`, `modal_app.py::merge_adapter --lora <repo@rev> --out /runs/... [--like <full ckpt>]`: the same
  fp32 W + delta rounded once as `--toward` at α 0, the LoRA's head and meta with `lora 0`, `weights "full"`; `--weights_dtype bf16`
  writes an fp32-trained Kev as bf16, one rounding after the fp32 merge, which is how the MLX full-weight path is checked on small models; Kev-27B at
  `/runs/r25-init/kev-27b-merged/checkpoint`, weights `f61fb0c5…`, ~6 min on 8 CPUs, `runs/r25-init/kev-27b-merged/merge.json`);