harness-evaluation

Files (17)

• SKILL.md
• openai.yaml · agents/
• scheduled-workflow.md · references/
• skill-evaluations.md · references/
• evaluation_state.py · scripts/
• basic.jsonl · scripts/fixtures/codex/
• branched.jsonl · scripts/fixtures/pi/
• followups.jsonl · scripts/fixtures/pi/
• resumed.jsonl · scripts/fixtures/pi/
• skill-injection.jsonl · scripts/fixtures/pi/
• tool-error-malformed.jsonl · scripts/fixtures/pi/
• unsupported-version.jsonl · scripts/fixtures/pi/
• recent_pi_sessions.py · scripts/
• recent_rollouts.py · scripts/
• test_evaluation_state.py · scripts/
• test_recent_pi_sessions.py · scripts/
• test_recent_rollouts.py · scripts/

SKILL.md (first lines)

# Harness Evaluation

Evaluate skills from real collaborative sessions. This is a personal field-
evaluation workflow, not a repo harness bootstrap and not a synthetic prompt
eval.

The evaluation criteria live here so repo-local and third-party skills do not
need to ship personal evaluator metadata. Read
[references/skill-evaluations.md](references/skill-evaluations.md) for named
skills in scope. When a skill is not cataloged, derive a temporary evaluation
contract from its description and instructions; do not add it to the catalog
unless the user wants to keep evaluating it.

Read the toolkit's
[skill prompting lens](../../../docs/skill-prompting-lens.md) before changing a
skill. When the finding concerns sidecars, handoffs, or orchestration, also read
the [Codex model workflow lens](../../../docs/codex-model-workflow-lens.md).

## Modes

### Targeted retrospective

Use when the user names session files, a session directory, a task, or a human
correction. Inspect the relevant active conversation branches completely enough
to reconstruct decisions before and after the correction.

### Daily scan

For a scheduled scan, read and follow
[references/scheduled-workflow.md](references/scheduled-workflow.md). It uses the
bundled SQLite state CLI to carry cursors, unreviewed findings, decisions, and
worktree review status across days.

For an ad hoc scan without persistent state, use the bundled helpers directly:

```bash
SKILL_DIR=<directory containing this SKILL.md>
python3 "$SKILL_DIR/scripts/recent_rollouts.py" --hours 24
python3 "$SKILL_DIR/scripts/recent_pi_sessions.py" --hours 24
```

Both scripts accept `--root`, `--since`, `--until`, and `--json`, use latest
user/assistant activity, and surface follow-up user messages so human
corrections are visible during shortlisting. Use `--until` when the reporting
window must remain stable while the scan runs. JSON output reports resumed
session counts. Report the matched session count. Inspect full session files
only after shortlisting.

Prioritize sessions with human correction, backing out or restarting work,
repeated attempts, unexpected skill activation, tool failures, recovery
messages, or substantial implementation/review work. These signals select
sessions for inspection; they do not show that the work or skill failed. Also
sample at least one apparently successful skilled session when practical so the
review is not based only on suspected problems.

The helpers locate evidence; their keywords and summaries are not evaluation
results.

## Reconstruct The Session

For each material session, capture:

- the user-visible goal and how it evolved;
- the repository and task boundary;
- skills injected, read, or conspicuously absent, including when they became
  active;
- important proposals, hypotheses, and implementation decisions, including the
  evidence that confirmed or rejected each material hypothesis;
- user corrections, rejected implementation work, restarts, and accepted
  tradeoffs;
- review findings and whether they exposed defects or expanded scope;
- relevant tool/interface failures and repository constraints;
- the resulting implementation or artifact when needed to judge the decision.

For Pi, follow the active `parentId` branch by default. For Codex, ignore
`session_meta` instruction noise. Treat injected skill bodies and available-
skill listings as attribution evidence, not as the user's actual request.

Do not infer that a skill was followed merely because it was loaded. Do not
blame a skill that was unavailable or activated only after the harmful decision.

Separate normal investigation updates from process or outcome corrections. A
plausible hypothesis that the agent checks and rejects is not rework, harm, or a
skill failure. It becomes material when the agent presents it as established,
acts on it without enough evidence, repeats it without gaining evidence, or does
not update its conclusion after contrary evidence.

## Evaluate Each Skill

Use the named criteria from the central catalog, plus these common dimensions:

1. **Activation** — correct trigger, false activation, or missed/late activation.
2. **Adherence** — which material instructions were followed, ignored, or
   impossible to apply.
3. **Decision effect** — what scope, fallback, architecture, test, review, or
   collaboration decision changed because of the skill.
4. **Human correction** — first determine whether the user corrected a material
   process or outcome problem, supplied new context, or confirmed a normal
   investigation update. For a real problem, assess whether the skill prevented,
   caused, or failed to prevent it.
5. **Cost** — context, questions, artifacts, review lanes, implementation state
   space, test burden, and review difficulty added.
6. **Counterfactual** — the strongest evidence for what likely would have
   happened without the skill. Keep confidence explicit.
7. **Owning layer** — personal skill, repo-local skill/docs, harness tool or
   schema, model behavior, environment, task-local context, or no durable change.

Classify the observed effect without a numeric score:

- `helped`
- `neutral`
- `harmed`
- `missed activation`
- `false activation`
- `insufficient evidence`

A good final result does not prove a skill helped. A failed command does not
prove a skill harmed. Look for the intervention's effect on decisions and human
correction.

From corymhall/toolkit, MIT.

More from corymhall/toolkit