rubric-eval

Files (37)

• SKILL.md
• SPEC.md
• judges.json · references/
• model_call.py · scripts/
• rubric-eval · scripts/
• rubric_eval.py · scripts/
• README.md · suites/architecture-v2/
• plan.txt · suites/architecture-v2/frames/
• review-bare.txt · suites/architecture-v2/frames/
• review.txt · suites/architecture-v2/frames/
• rubric.json · suites/architecture-v2/
• scenarios.json · suites/architecture-v2/
• README.md · suites/architecture-v3/
• plan-ref.txt · suites/architecture-v3/frames/
• plan.txt · suites/architecture-v3/frames/
• review-bare.txt · suites/architecture-v3/frames/
• review-ref.txt · suites/architecture-v3/frames/
• review.txt · suites/architecture-v3/frames/
• rubric.json · suites/architecture-v3/
• scenarios.json · suites/architecture-v3/
• README.md · suites/architecture-v4/
• plan-ref.txt · suites/architecture-v4/frames/
• plan.txt · suites/architecture-v4/frames/
• review-bare.txt · suites/architecture-v4/frames/
• review-packet.source · suites/architecture-v4/frames/
• review-packet.txt · suites/architecture-v4/frames/
• review-ref.txt · suites/architecture-v4/frames/
• review.txt · suites/architecture-v4/frames/
• make_packet_frame.py · suites/architecture-v4/
• rubric.json · suites/architecture-v4/
• scenarios.json · suites/architecture-v4/
• README.md · suites/architecture/
• plan.txt · suites/architecture/frames/
• review-bare.txt · suites/architecture/frames/
• review.txt · suites/architecture/frames/
• rubric.json · suites/architecture/
• scenarios.json · suites/architecture/

SKILL.md (first lines)

# rubric-eval

Read [SPEC.md](SPEC.md) before an experiment; it is the reference for the
condition rules, models, judge, statistics, ship rule and change lifecycle.
This page is how to drive it.

The CLI is `scripts/rubric-eval` (Python 3, no dependencies). Each subcommand is
one step with a file contract, so other skills can call any step:

```sh
E=scripts/rubric-eval
$E extract skills/shiploop/scripts/shiploop_prompts.py --symbol INTERACTION_DESIGN
$E build RUN --arm base=OLD.py::INTERACTION_DESIGN --arm cand=NEW.py::INTERACTION_DESIGN --trials 2
$E run RUN --model grok             # execution: grok-4.7, medium effort (sonnet for exploratory rounds)
$E judge RUN --model opus           # the one judge for the round: claude-opus-5-5, medium effort
$E analyze RUN --baseline base      # percent-of-points scores, paired tests, quality > tokens > time
$E reliability RUN --model opus --n 30   # kappa and noise when the judge changes; record in references/judges.json
echo "prompt" | $E call --model grok
```

Review experiments grade a review's revised plan: build with `--frame review
--plans-from PLAN_RUN --plans-arm ARM` and one `--arm` per review focus.

## Workflow for a change

1. State the hypothesis and what the change might harm.
2. Extract arms from source (never retype them), then build, run, judge and
   analyze. Keep every arm under one condition (SPEC section 5).
3. Run `adversarial-review` on the change and its evidence, with a model family
   other than the one proposing it.
4. Turn each testable high or medium finding into an experiment with its
   refuting result stated first; run it; revise the change if a finding holds.
5. Rerun on the exact text that will ship. Ship only when `analyze` names the
   arm as `winner` there and the loop is complete (SPEC section 8). Quality
   decides first; tokens, then time, decide only between equivalent arms.
6. Record everything in the results doc (SPEC section 10).

Long runs:
- Name `--model` on every `run` and `judge` command, keep one grading loop per
  round, and confirm from the process list that a stopped loop is gone
  (SPEC section 11).
- Report progress at the interval the user asks for, as a narrative: where the
  work sits in the larger goal, what finished and showed, what is running and
  why, the conclusions so far with their strength, the learnings, and what
  comes next. Interim paired results stabilise as more cells finish.

From whichguy/skill-craft, MIT.

More from whichguy/skill-craft