eval-answer

Files (31)

• SKILL.md
• coverage-limits.md · reference/
• definition-ledger.md · reference/
• ledger-schema.md · reference/
• check_contamination.py · scripts/
• check_contamination_test.py · scripts/
• check_coverage.py · scripts/
• check_coverage_test.py · scripts/
• check_findable.py · scripts/
• check_findable_test.py · scripts/
• check_must_not_use.py · scripts/
• check_must_not_use_test.py · scripts/
• config.py · scripts/
• config_test.py · scripts/
• golden_rows.py · scripts/
• golden_rows_test.py · scripts/
• init_truth_package.py · scripts/
• init_truth_package_test.py · scripts/
• json_scan.py · scripts/
• ledger.py · scripts/
• ledger_test.py · scripts/
• mcp_payload.py · scripts/
• mcp_payload_test.py · scripts/
• publisher_rest.py · scripts/
• publisher_rest_test.py · scripts/
• score_retrieval.py · scripts/
• score_retrieval_test.py · scripts/
• verify_definitions.py · scripts/
• verify_definitions_test.py · scripts/
• verify_goldens.py · scripts/
• verify_goldens_test.py · scripts/

SKILL.md (first lines)

# Evaluate One Answer

One user intent, answered once. This skill decides whether that answer was
correct, records the evidence, and stops.

**Scope boundary:** verdict and events only. No diagnosis, no model edit.

## The unit

A chat is not the unit. Segment by user intent. Feedback ("break it out by
region") is a revision inside the same answer; grade the final accepted
revision.

Take the question from the stored case (`evals/<set>/cases.jsonl`), never from
memory or a truncated console line. Record `question_sha` of the exact text
the answerer saw. Record `servedRevision` from `get_context` or reload, not
the package name: a same-named decoy has been measured for hours.

## Step 1: Contamination check, before any score

The answerer can Read or Shell its way to gold. Publisher traces do not see
that, so the check runs on the HOST-side tool-use log you kept for the
answerer subagent (every tool name and its path or command), plus the MCP
call counts the answerer reported.

The checklist. An attempt is contaminated when its log shows any of:

1. a Read, Shell, or any file tool touching `evals/` or a gold artifact path;
2. any access to the model file under test through a file tool (the
   `modelPath` argument on an MCP `execute_query` is NOT contamination; the
   server resolves it, the answerer never reads the file);
3. `reported_calls` greater than `host_tool_uses` (the detectable
   under-report floor is reported at most total tool uses). `host_tool_uses`
   is EVERY tool use the host logged, MCP calls included; while it counted only
   the non-MCP ones this comparison was true of almost every clean attempt, so
   a run whose attempts predate the split cannot be checked this way.

`skills/eval-answer/scripts/check_contamination.py` is a reference aid that
mechanizes the same checklist over a JSON log; your reading of the transcript
is the check, the script is a second pair of eyes.

Contaminated attempts get `verdict: null` and `contaminated: true`. They are
excluded from the run aggregates. They are not "wrong answers."

If you cannot produce a host log, mark `contaminated: "unknown"` on both the
attempt and its score event, and do not treat the attempt as a clean pass.

## Step 2: Re-run the submitted query yourself

Never score the agent's reported rows. Take its final query, execute it with
`execute_query`, and write a prediction CSV under the run's
`artifacts/` directory.

A named view is a submitted query. `execute_query` takes either ad-hoc Malloy
or a `queryName` plus `sourceName`, and its own tool description steers an
answerer to the named form; record it as the Malloy it stands for,
`run: <source> -> <view>`, which re-executes and reads the same as any other.
Capturing only the ad-hoc form recorded an attempt that did query as
`submitted: false` with no query to re-execute, and the judge then graded prose.

`submitted: false` when there is no final query. That is not a wrong answer, and
it is not by itself a reason to withhold a verdict. No verdict can be issued
(`verdict: null`, with the reason) when the attempt produced neither a query nor
any answer text, when the golden is missing, provisional, invalid, or ambiguous,
or when a verified golden that holds a value has no local artifact to compare.
A `criteria` golden holds no value and needs no artifact: its clauses are the
whole comparison, and withholding a verdict for a missing artifact there drops
a scorable case out of the pass rate. An attempt that
wrote prose and ran nothing IS judged: against a golden holding a value, an
answer containing none of it is `no_match` however well it reasons.

## Step 3: Judge the answer

Spawn one fresh judge subagent per attempt, following
`skill:eval-judge` (the rubric, the anchors, and the output shape live
there; this skill does not restate them). Give it the question, the golden,
your re-executed prediction rows, the canonical query when present, and the
relevant source and field definitions from the model. It returns
`{verdict, confidence, why, column_pairing}`.

- The judge sees gold. It is therefore never the answerer, and its verdict
  never leaks back to any answerer.
- Confidence 5 or lower records as `needs_human`: neither a pass nor a fail,
  excluded from acceptance arithmetic, queued for a human look.
- `near_match` is also neither. It means defensibly different, not "nearly a
  pass", and it stays out of the pass rate and the acceptance check for the same reason
  `needs_human` does. Report the count; do not fold it into either column.
- When a human overrules a verdict, append the case to
  `evals/<set>/judge-regressions.jsonl`.
- For a scalar golden, the same protocol applies to a one-value prediction.
  For `unanswerable`, a refusal that names the gap is the pass; a confident
  numeric answer is the fail.
- Large row sets are still the judge's job. There is no scripted row oracle:
  a script that can pass a wrong answer is worse than none, and the rubric's
  containment and column-pairing rules are what the comparison needs.
- `golden.mustNotUse` is the exception, and it is not the judge's. It names the
  similar-but-wrong field, and using one is a failure however good the number
  looks, which is a question about query TEXT. Run
  `scripts/check_must_not_use.py` over the final query: a named field found
  there forces `no_match` and records `must_not_use_hits`, keeping the judge's
  own verdict beside it as `judge_verdict`. Only a BARE name vetoes. An entry
  with a connective (`weekly_active_users as a cumulative series`, `product.cost
  through the order_items join`) objects to a use of the field rather than to
  the field, so it goes to the judge as prose, as does an entry that names no
  field at all and a path's bare leaf. A veto that fires on a correct answer is
  worse than one that misses, and reading the head of `X as ...` as "ban X"
  failed an answer whose only sin was showing X as an extra column.

## Step 4: Score what retrieval delivered

Per attempt, mechanically, from the ledger -- `scripts/score_retrieval.py`. Each
case names the entities its answer depends on (`expectedEntities.required`, and
`requiredAnyOf` groups where the model offers more than one route). An entity
was delivered if the attempt's `get_context` calls returned it as a ranked entity
under its id, under the same type and name on a sibling source, or by name inside
a returned source's documentation -- text the answerer reads and acts on. Only
`missing` is a retrieval miss; the route per entity is recorded so the strict
count is still there.

From malloydata/publisher, MIT.

More from malloydata/publisher