eval-import

Files (4)

• SKILL.md
• case-format.md · reference/
• import_cases.py · scripts/
• import_cases_test.py · scripts/

SKILL.md (first lines)

# Import questions into an eval set

Questions arrive in whatever shape their author had. This turns them into
`evals/<set>/`, ready for a run, and records what each key is actually worth.

**Scope boundary:** cases only. This never answers a question, scores an
answer, or edits a model. It is `skill:eval-loop`'s scrape step, expanded,
and it is where the set's honesty is decided.

The file shapes are defined once, in `reference/ledger-schema.md` of
`skill:eval-answer`. Read them there; this skill does not restate them.
`reference/case-format.md` covers what is specific to an import: which fields
an arriving item maps to, and the four physical formats.

## Where the set goes

Two separate rules, and "package" blurs them:

- **The same git repository as the model it evaluates**, so one commit pins
  the model and the ledger together. That is what a checkpoint is.
- **Never inside the directory tree the answerer's package serves.** The
  answerer can read files in the served tree, and gold there is a
  contamination path.

So the set is a SIBLING of the package under test, in its repo. A repo with
`ecommerce/` under test gets `evals/ecommerce-questions/`, beside it.

## Step 1: keep the file as it arrived

Copy it verbatim into `evals/<set>/as-received/`, and never edit that copy.
Not the encoding, not the line endings, not an obvious typo.

It is there so a later disagreement is about the key and not about whether
somebody transcribed the question correctly. That disagreement is guaranteed:
`skill:eval-loop`'s `reference/auditing-an-answer-key.md` is about what happens
when it arrives, and its first check is the question against this file.

If the source cannot be committed, because it carries names, addresses,
internal URLs, or row values somebody pasted into a thread, say so. Record in
`set.json` where the original lives and a hash of it, note under
`sourceNote` that the copy is withheld and why, and tell the user that the
byte-for-byte check is now impossible for anyone without the original. Do not
redact silently: a redaction you got wrong becomes the provenance record.

## Step 2: classify each item by what arrived WITH the question

This is the whole job, and getting it wrong is how an unverified number gets
scored as a fact.

| what arrived | write | scorable on day one |
|---|---|---|
| the question only | a case with **no `golden`** | no. It measures coverage and discoverability, which is a real measurement |
| question + their query + a number | `golden.value`, their query as `canonicalQuery`, `verifiedBy: authored_query`, `status: provisional` | after step 3 agrees |
| question + a number, no query | `golden.value`, `status: provisional`, no `verifiedBy` | no. Somebody typed a number |
| question + prose criteria | see step 4 | depends on what the criterion is |

Two rules over the whole table:

**No golden holding a VALUE imports as `verified`.** `verified` means two
differently shaped derivations agreed through the truth package, and an import
has performed neither. `provisional` goldens are unscorable by design:
`skill:eval-answer` issues `verdict: null` on them. That is not a gap to work
around.

It is also not a dead end, and it used to read like one. The way out is
`verify_goldens.py --promote`, which marks a `provisional` golden `verified`
once its value re-derives cleanly from the truth package and a second
derivation exists. It is the only thing anywhere that writes `golden.status`.
Tell the user that command when you hand over a set of provisionals, because
until it is run the set scores nothing. A set that arrives with 60 typed numbers genuinely has 60 unverified
numbers, and the number it can honestly produce on day one is a coverage score.

The exception is a golden that holds no value at all, and there are two
kinds. Prose criteria (`kind: criteria`, step 4) have nothing to re-derive:
the author's words ARE the key. So does a case whose stated answer is that the
data is not there (`kind: unanswerable`), where the pass is a refusal that
names what is missing and a confident number is the failure. Both import
`verified` with `verifiedBy: authored_criteria` and score immediately. The
rule is about numbers, because a number is the thing that can be wrong while
looking right.

Watch for the second one in what arrives. A criterion reading "a refusal that
names the missing data" is not a rubric clause on a normal case, it is the
whole key, and importing it as `criteria` on a case with a value would make a
refusal fail.

**A question with no golden is a case, not a reject.** Do not hold questions
back until somebody derives keys. A set of bare questions already measures
whether the model can express an answer at all, and the answers it produces
are what the keys get derived from.

### Deriving a key yourself, when nothing arrived with the question

The table above is about what ARRIVED. Sometimes nothing did, nobody knows the
right number, and the set still needs keys. You may derive one, and the rules
are the ones already stated rather than new ones:

- Use your full model access and query until you can defend the number. The
  agent that gets tested later holds only `get_context` and `execute_query`;
  you are not that agent, and the gap is the point.
- Record the query as `canonicalQuery`. A number with no query is somebody's
  guess, which is the row above it in the table.
- Record the entities you used to get there as `expectedEntities.required`, in
  the `kind:source:name` form: the measures, dimensions and views the answer
  cannot be produced without. `reference/case-format.md` says to leave this
  field empty at import because guessing it invents a retrieval expectation
  nobody stated; you are not guessing, you just used them. That is the only
  trustworthy producer this field has, and it is what makes retrieval
  measurable on a set nobody sent keys for. Where the model offers two
  legitimate routes, write a `requiredAnyOf` group rather than pick one.
  `verify_goldens.py` check 5 audits every id against the model, so a wrong one
  is a hard finding rather than a silent retrieval miss.
- **Derive the list with the model in front of you, not from memory, and not
  with grep.** The model knows what each field IS; an author does not reliably.
  Ask the question of the model: which entities does an answer to this depend
  on, and is each a measure or a dimension. Confirm every id with `get_context`
  before writing it. A list written from recollection is the most common way a
  set acquires a retrieval expectation nobody can satisfy.

From malloydata/publisher, MIT.

More from malloydata/publisher