flow-eval-run

Files (1)

• SKILL.md

SKILL.md (first lines)

# /flow-eval-run

A **thin router** for the eval loop. It owns nothing itself — it points at the two
systems that do the work, and links the authoritative docs instead of restating them.

## The two routes (don't reinvent either)

- **Peer orchestration → `/flow-pair`.** Spawning/steering the subject (and any
  reviewer) peer, delegating a bounded packet, harvesting learnings — that is
  `/flow-pair`'s job. Invoke it; do not hand-roll `pij` orchestration here.
- **Scoring → `harness flow-eval` + its briefing.** The evaluator is the
  `flow-eval` extension. Its authoritative doc is
  [`.harness/extensions/flow-eval/instructions.md`](../../../.harness/extensions/flow-eval/instructions.md)
  — read it for lane/axis/verdict semantics. Two actions:
  - `scaffold --slug <slug>` — writes a ready-to-edit scenario skeleton under
    `live-testing/scenarios/<slug>/`.
  - `score --scenario <slug> --session <pij-id> [--worktree <path>]` — grades a
    **finished** session and writes `report.{json,md}`. It never drives pij.

## Two committed scenario variants — pick the question first

| Scenario | Packet | Measures | Base ref |
|---|---|---|---|
| `md-to-pdf` | fully blind (no method) | **ambient adoption** — does the subject reach for the flow unprompted? | old tag (capability-only; telemetry lane blind by design) |
| `md-to-pdf-flow` | `/the-flow` MANDATED, only that | **flow fidelity + stage economics** — is the flow driven honestly, what does each stage cost? | current-code sha (telemetry lane fully live; harness-loop conduct scored deterministically) |

The variant decides the drive style too: the mandated variant takes **no stage
nudges** (the subject's own cadence is scored) and the orchestrator answers flow
gates under the **fixed gate-conduct policy** in its `prompts/orchestrator.md`.

### `builder-team-md-to-pdf` — delivered method and capability

Use the versioned, isolated packaged-consumer route in
[`docs/how/flow-conformance-eval.md`](../../../docs/how/flow-conformance-eval.md#blind-builder-team-cohort-version-1).
It owns preparation, native positive control, score bindings and retained evidence.
Do not use the historical worktree/telemetry recipe below for this cohort.

Deliver only the scenario's `prompts/subject.md` plus root/reply instructions.
Builder is mandated; decomposition, stages and checks are not prescribed. Keep
assertions, the evaluator runbook and reports out of every subject/worker root.
Use `--evidence-source native --subject-plan <new plan.dd.json> --base-ref <consumer
preparation SHA>` from the evaluator root. Native Flowspace turns are not telemetry
segments. Requested/observed configuration is not provider attestation. Require
working PDF/Mermaid output, independent composed-SHA review and preserved evidence;
the process score alone never decides acceptance.

### `dd-native-builder` — do the GATES do any work?

| Scenario | Packet | Measures | Base ref |
|---|---|---|---|
| `dd-native-builder` | `/builder` MANDATED, only that | **discoverability + gate efficacy** — can a subject that has never seen the dd-native flow find it, and does anything ever REFUSE the subject? | current-code sha (must carry the `command_exit` error-code capture, or `gate-refused` cannot be answered) |

Two things to get right when running it:

- **Resolve A11**: its `cmd` is the placeholder token `SUBJECT_PLAN_VALIDATE_COMPLETE`.
  Pass `--resolve SUBJECT_PLAN_VALIDATE_COMPLETE="node harness/cli/bin/harness.js plan
  validate <the subject's plan.dd.json> --complete"`. Unresolved it scores `unknown`,
  which is honest but wastes the row.
- **Dry-score before spending a subject**: `harness flow-eval score --scenario
  dd-native-builder --session <a session that does not exist>` and check every
  placeholder row reads `?`. A placeholder must be the WHOLE `cmd`; an embedded
  `${VAR}` is executed literally and false-FAILS. This bundle was authored with that
  mistake and a dry score caught it.

Read A10 (`gate-refused`) first, not the total. A run that scores well while A10 fails
means the subject was never stopped — which tells you about the gates, not the subject,
and is the finding the scenario exists to produce.

## The loop (≈6 beats)

1. **Pick or scaffold a scenario** under `live-testing/scenarios/<slug>/`
   (`harness flow-eval scaffold --slug <slug>`), then **hygiene**: remove stale
   eval worktrees/branches (`git worktree list` → remove + prune) — a fresh
   subject will correctly *resume* leftover plans it finds (F12).
2. **Cut the subject worktree at `scenario.json → base.ref` yourself**, then
   **spawn the subject peer from INSIDE it** — `(cd "$WORKTREE" && pij spawn …)`
   — so the shared main checkout is mechanically out of reach (a packet
   instruction alone failed 1-in-4 live). Canary-verify the model (footer +
   reply-over-pij), then deliver the stripped packet + worktree line + the
   standing comms contract (control-plane peers push nothing unless told:
   append "reply via `pij send <orch-id>`" to EVERY send — it does not survive
   the subject's compactions).
3. **Let it run** to completion in its worktree; capture its `pij` id. Answer
   task-scope questions and flow gates per the runbook's gate-conduct policy;
   never nudge toward scored behaviors.
4. **Score it**: `harness flow-eval score --scenario <slug> --session <pij-id> --worktree <path>`
   (add `--subject-harness/-model/-effort` + `--base-ref <ref>` when the real subject/base
   differs from `scenario.json`'s defaults; add `--resolve <id>=<command>` for each
   subject-specific placeholder `command-succeeds` assertion — resolve per-run, never edit
   the committed `assertions.json`; the placeholder ids are per-scenario — `A7`/`A8` in
   `md-to-pdf`, `A8`/`A9` in `md-to-pdf-flow` — read the scenario's own runbook).
5. **Read the report** at `.harness/live-testing/<slug>/<run-id>/report.{json,md}`;
   fill any `judged` fields, note the two-axis scores + any `mimicry` alarm. For
   fidelity-class judged fields, cross-check the flight plan + telemetry against
   the subject's raw session events (skill capture can gap on sparse-command
   copilot sessions — an A-skill fail is honest-per-evidence, not always
   subject fault) and separate new-since-base artifacts from inherited ones
   (`git status` in the worktree).
6. **Re-render after filling judged fields**:
   `harness flow-eval render --scenario <slug> --run <run-id>` regenerates `report.md`
   from the filled `report.json` (idempotent; no telemetry; the ledger is append-only).
   Then, before teardown: **sync telemetry FROM the worktree** (`harness telemetry
   sync` there — the buffer is per-clone and dies with the worktree), kill any
   process still holding the worktree (headless Chrome, vitest), `pij close` the
   subject, and `git worktree remove --force` + branch delete + prune.

## One worked invocation (md→PDF)

```
harness flow-eval score --scenario md-to-pdf --session pij-abc123 --worktree /path/to/subject/worktree --resolve A7='node harness/cli/dist/index.js <new-verb> --help' --resolve A8='<pdf-validator-cmd>'
# → .harness/live-testing/md-to-pdf/<run-id>/report.{json,md}
harness flow-eval render --scenario md-to-pdf --run <run-id>   # re-render report.md after filling judged fields
harness flow-eval supersede --scenario md-to-pdf --run <stale-run-id> --by <corrected-run-id>  # only if you re-scored the session
```

## Running the `harness` bin

- **In THIS repo** (the harness's own home), call the bin directly:
  `node harness/cli/bin/harness.js flow-eval …` (per `AGENTS.md` — don't `npx` the

From AI-Substrate/harness-engineering, MIT.

More from AI-Substrate/harness-engineering