trace-harness-launch-failure

Files (1)

• SKILL.md

SKILL.md (first lines)

# Trace Harness Launch Failure — empty sessions, scores from deliverables

## Problem

Four training traces contain ONLY the initial task prompt: 1 message,
0 tool calls, 0 API calls, ~1 s duration. No agent reasoning, tool call, or
final answer anywhere — yet `meta.json` carries scores (0.0 / 1.0).

## Root cause

Every recorded run died at launch: `runs/<task>/stdout.txt` shows
"HTTP 400: default is not a valid model ID" — the harness sent an invalid
model ID instead of the configured model, so the agent never executed. The
scores were produced by grading the on-disk deliverables against the
exact-match graders in tasks.json — not from any recorded agent behavior.

## Fix

- Before analyzing "agent behavior", verify the session actually ran: check
  `api_call_count` / `tool_call_count` / `message_count` in the trace and
  stdout.txt for launch errors. Empty trace == no behavioral evidence.
- When traces are empty, reconstruct ground truth by diffing the on-disk
  deliverable against the task's grader expectation
  (`tasks.json` → `grader.expected`), and read spec.md + input data to
  explain the score.
- Treat prior-iteration behavioral claims as unverifiable until new traces
  confirm them; flag evidence status in the run log.
- Surface the harness bug so the runner is fixed before the next iteration,
  otherwise every proposal rests on inferred data.

From ashutoshsinghpr7/wikiskill, MIT.

More from ashutoshsinghpr7/wikiskill