eval

Files (1)

• SKILL.md

SKILL.md (first lines)

# Eval

`uv run eval` evaluates a model in one or more environments and exits after one epoch per source. It reuses the orchestrator's eval pipeline: one env server per source, a concurrency band, every episode through the monitors, and a trace stream that `--resume` continues from. Online evals of training runs are the `training` skill.

## Start an eval

The user launches runs; hand over the command unless told otherwise. Always cap smokes (`-n`, `-r`).

```bash
uv run eval gsm8k -n 32 -r 4                                       # Prime Inference
uv run eval gsm8k -n 32 -r 4 -c 8 --env.agent.harness.id bash      # pin concurrency; set a field of the env block
uv run eval gsm8k -n 32 -r 4 -m Qwen/Qwen3-4B --client.base_url http://localhost:8000/v1   # a local `uv run inference` server
uv run eval @ eval.toml --run.name my-eval                          # multi-source TOML
uv run eval @ eval.toml --run.name my-eval --resume                 # continue an interrupted run
uv run eval @ eval.toml --dry-run                                   # resolve and write the config, exit
uv run eval @ eval.toml --no-monitors.prime                         # keep the run off the platform (on by default)
```

- Config: `EvalConfig` (`packages/prime-rl-configs/src/prime_rl/configs/eval.py`); `uv run eval -h` lists the fields.
- Entrypoint: `src/prime_rl/entrypoints/eval.py`; implementation `src/prime_rl/eval/eval.py`, shared engine `src/prime_rl/eval/runner.py`.
- Shorthands (single-source runs): `<taskset-id>`, `--env.<field> <value>`, `-n` `select.limit`, `-s` `select.shuffle`, `-r` `group_size`, `-m` `model`, `-c N` pins `concurrency.min_inflight = max_inflight = N`. They cannot be combined with a TOML that defines `[[source]]` blocks.
- Concurrency: an API exposes no vLLM `/metrics`, so run pinned there; against `uv run inference` set `min_inflight < max_inflight` in `[concurrency]` to adapt to KV usage.
- Env servers: spawned per source on OS-assigned ports (published to `configs/attempt_N/resolved/envs/eval/<name>.address`) unless the source sets `serve.address`.
- Ready-made configs: `configs/debug/eval/*.toml`, one per shape (single turn, multi turn, resume, multi env, aime2026, tb2).

Minimal multi-source TOML (the eval block is flattened to the top level; per-source `select`, `group_size`, `sampling` override the top level):

```toml
model = "Qwen/Qwen3-4B"
group_size = 4

[select]
limit = 32

[client]
base_url = "http://localhost:8000/v1"

[[source]]
env.taskset.id = "gsm8k"
env.agent.harness.id = "bash"

[[source]]
env.taskset.id = "aime25"
env.agent.harness.id = "null"
env.agent.runtime.type = "subprocess"
```

## Select tasks

`select` picks which tasks of a source's taskset run. The steps apply in a fixed order: `include` and `exclude` (by `idx` position, `ids`, `keys` or `names`), then `shuffle` (under `seed`, default 0), `skip`, `limit`. Set it once for every source of a group (`[select]` in an eval TOML, `[orchestrator.eval.select]`, `[orchestrator.train.select]`), or per source (`select.limit = 50` in a `[[source]]`); a source's own fields win.

```bash
uv run eval gsm8k -n 50                                   # the first 50 tasks
uv run eval gsm8k -n 50 -s                                # a random subset of 50 (same tasks every run)
uv run eval gsm8k -n 50 -s --select.seed 1                # another random subset
uv run eval gsm8k --select.include.idx 0:10,42            # tasks by position (ints and Python slices)
uv run eval terminal-bench-2 --select.include.names '["terminal-bench/fix-git"]' # tasks by name; `ids` and `keys` work the same
uv run eval terminal-bench-2 --select.exclude.keys '["<task.key>"]' # drop known-broken tasks (keys are on every trace)
```

Disjoint train/test splits from one taskset:

```toml
# contiguous: eval on the first 100 tasks, train on the rest
[[orchestrator.eval.source]]
env.taskset.id = "gsm8k"
select.include.idx = [":100"]

[[orchestrator.train.source]]
env.taskset.id = "gsm8k"
select.exclude.idx = [":100"]

# random: the same shuffle on both sides; eval takes 100, train skips them
#   eval:  select.shuffle = true, select.limit = 100
#   train: select.shuffle = true, select.skip = 100
```

The shuffle comes before `skip` and `limit`, so a larger `limit` extends the same selection: resuming a `-n 50 -s` run with `-n 100` keeps the 50 landed tasks and adds 50 more.

## Monitor an eval

Run dir: `output_dir / run.name` (auto `<envs>--<model>--<short-id>`; `ls -t outputs | head -1` finds the latest). The console stays quiet while the eval runs; everything is in the files and the dashboard (`dashboard` skill).

```
{run_dir}/
├── configs/latest/            # command.txt, the launch TOML, resolved/eval.json
├── logs/latest/
│   ├── eval.log               # the eval process
│   └── envs/eval/{name}.log   # one log per env server
└── monitors/file/             # metrics.jsonl, the trace stream, traces/live/ (one file per live trace), plan.json
```

```bash
tail -F {run_dir}/logs/latest/eval.log
grep -E "WARNING|ERROR" {run_dir}/logs/latest/eval.log {run_dir}/logs/latest/envs/eval/*.log
grep SUCCESS {run_dir}/logs/latest/eval.log                            # one "Evaluated <env> ... Reward 0.xxxx" line per source
uv run python -m prime_rl.monitors.file.traces {run_dir} [<trace_id>]   # live rollouts by phase, or one assembled live trace
```

The progress line in `eval.log` counts live rollouts by phase (`- boot 1 · running 3`). A rollout stuck in `boot` for minutes is waiting on its sandbox; one in `running` with a frozen turn count is waiting on a model call or tool.

Metrics live under `eval/<env>/all/<agent>/…` (`reward/mean`, `is_truncated/mean` — raise `sampling.max_completion_tokens` when high, or score cut-off runs as 0 with `env.agent.scoring.skip_on_truncated` / `skip_on_timeout` (`env.agent.max_turn_tokens` caps each response and stops the run once one hits it, even in harnesses like RLM that continue past a capped turn), `has_error/mean`, the taskset's own metrics). Validate a result by reading a few traces in the dashboard rather than trusting the mean alone.

Stop a run with SIGINT/SIGTERM to the eval PID (`ps aux | grep PRL::Eval`); `--resume` restores the landed episodes from the trace stream and runs only the rollouts still owed. Any config may be overridden on resume (`select`/`group_size` change what is owed); keep the model, sampling and env config the same to keep the landed episodes comparable. Env servers are children of the eval process and exit with it.

From PrimeIntellect-ai/prime-rl, Apache-2.0.

More from PrimeIntellect-ai/prime-rl