voice-eval

Files (30)

• SKILL.md
• prompts-20x20.json · assets/
• conftest.py
• pyproject.toml
• __init__.py · scripts/
• arms.py · scripts/
• ballot.py · scripts/
• detector.py · scripts/
• drift.py · scripts/
• floor_gate.py · scripts/
• human_study.py · scripts/
• prompts.py · scripts/
• report.py · scripts/
• sibling_skills.py · scripts/
• signal_deltas.py · scripts/
• winrate.py · scripts/
• skill_api.py
• __init__.py · tests/
• test_arms.py · tests/
• test_ballot.py · tests/
• test_detector.py · tests/
• test_drift.py · tests/
• test_floor_gate.py · tests/
• test_human_study.py · tests/
• test_prompts.py · tests/
• test_report.py · tests/
• test_sibling_skills.py · tests/
• test_signal_deltas.py · tests/
• test_smoke.py · tests/
• test_winrate.py · tests/

SKILL.md (first lines)

# voice-eval

The HFR v2 acceptance test. Deterministic Python helpers do the bookkeeping and
statistics; **the running model does the generation and the judging in-session** (no
API key), exactly like `triadic-voice`/`triadic-voice-v2`. v1 is the frozen control.

## How to run the 20×20 (in-session)

1. **Load prompts.** `load_prompts()` → 20 prompts stratified 7/7/6.
2. **Generate both arms.** For each prompt, write one v1 passage (follow
   `triadic-voice/SKILL.md`) and one v2 passage (follow `triadic-voice-v2/SKILL.md`,
   using `build_generation_brief(topic, rotation)`). Feed these as the `generate_v1` /
   `generate_v2` callables to `run_arms(...)` → 40 passages.
3. **Gate on the v1 floor.** `gate_passages(passages, battery=default_battery)`. Any
   passage with violations is **regenerated** before scoring, so both arms are equally
   floor-clean (REQ-VEVAL-010).
4. **Signal deltas.** `compute_deltas(passages, scorer=default_scorer, signals=SIGNAL_NAMES)`
   (import `SIGNAL_NAMES` from the `liveliness-signals` skill_api — the 8 signal names)
   → per-signal mean delta v2−v1, overall and per register (REQ-VEVAL-011).
5. **Blind pairwise judge.** `build_ballots(v1_passages, v2_passages)` → 40 ballots
   (each pair judged in both orders, length-matched). For each ballot, judge blind:
   read A and B, give a chain-of-thought rationale, then fill the verdict
   (`keep`, `want_next`, and ordinal `momentum/clarity/voice_authority/readability/
   trustworthiness`). Never look at the arm labels. Aggregate with
   `win_rate(filled_ballots, target="v2")` → win-rate + CI (REQ-VEVAL-012).
6. **Formula-drift.** Within each arm, `arm_drift(passages, struct_of=…, analogy_of=…)`
   using `default_struct_tokens` for the POS skeletons (REQ-VEVAL-013).
7. **(Optional) detector** — advisory only, never gates (REQ-VEVAL-014).
8. **Report.** `evaluate_success(...)` then `render_report(...)`. v2 passes only when it
   is floor-clean, scores higher on the positive signals, wins >50% pairwise,
   trustworthiness is not worse, and drift is lower (REQ-VEVAL-015).

## Human study (later run)
`build_study(pairs, seed, rubric)` builds the ≥50–60-item blind A/B scaffold;
`fleiss_kappa(...)` measures inter-rater reliability; `graduate(...)` is the gate that
promotes a signal from advisory to gating only on a moderate positive Spearman
correlation whose CI excludes zero, never degrading trustworthiness (REQ-VEVAL-016/017).

## Helpers
`scripts/prompts.py`, `arms.py`, `floor_gate.py`, `signal_deltas.py`, `ballot.py`,
`winrate.py`, `drift.py`, `detector.py`, `report.py`, `human_study.py`,
`sibling_skills.py`. All deterministic and unit-tested; siblings loaded repo-first.

From CharlesHoskinson/russellian-book-suite, MIT.

More from CharlesHoskinson/russellian-book-suite