eval-improve

Files (4)

• SKILL.md
• output-contract.md · reference/
• improve.py · scripts/
• improve_test.py · scripts/

SKILL.md (first lines)

# Improve the Model

Takes an issue with `owner: model` or `owner: package-skill` and produces **one
smallest edit** that closes the gap. Every factual claim is backed by a query
you ran.

The two owners license different edits and are never interchangeable:

- **`model`** changes what the data means or how it is described:
  a `.malloy` file.
- **`package-skill`** changes what the agent is told about using this package:
  a file under the package's `skills/`. Use it for a rule that spans entities or
  is not a property of any one of them. A fact about a single field belongs in
  that field's `#(doc)`, which reaches every agent whether or not it loaded a
  guide.

Both live in the model repo, so one git commit checkpoints and rolls back
either.

**Two hard boundaries:**

1. **No diagnosis evidence, no edit.** If the issue cannot name a concrete gap
   with a trace or probe, record that and stop. Edits from an empty diagnosis
   have been the inert and wrong ones.
2. **This skill never accepts its own edit.** You propose and verify. The
   acceptance check
   in `skill:eval-loop` admits or reverts. An improver writing the query it
   already knows proves the fix is possible, not that the next blind agent
   will find it.

## Step 0: What the evidence entitles you to change

| Evidence | Edits permitted |
|---|---|
| Verified golden, or a user who states the answer | Any tier. Probes required. Check the golden first. |
| Wrong answer, then a corrected one the user accepted | Prefer docs over structure. The diff between attempts is the missing knowledge. |
| User accepted, later contradicted | Docs, labels, index only. No structural change. |
| Doubt only, or retrieval-only (no verdict) | Docs, labels, index only, and only where the transcript shows a concrete confusion. |
| Silence | **No edit.** |

A `package-skill` edit sits at the docs tier throughout: it is permitted
wherever the table permits docs, and never on silence. It is the easiest artifact
in this loop to overfit, because it is prompt text sitting next to the question
it just failed, so the expert test in Step 1 applies to it word for word.

Do not edit for `BAD-REFERENCE` or `AMBIGUOUS-REFERENCE`. Those are the
golden side door in `skill:eval-loop`: repair or hold the golden, bump
`goldenRevision` on the case, and open a new baseline run. Being right and unmatched
beats encoding a defect or an unsettled key. Do not edit for an `agent-skill`,
retrieval, or dataset owner: `agent-skill` means the shared skill, which is not
in this repo, and widening it by editing one package's copy hides the gap from
every other package.

## Step 1: What a correct answer may teach

Encode what a domain expert would volunteer unprompted: systems of record,
vocabulary to stored codes, what a metric means and at what grain, which
relationship is the real one.

The expert test, per edit: *would a domain expert have said this about their
data with no question in front of them?* Reject:

- a field that hard-codes this question's filter and serves no other question
- this question's text, qid, or expected numbers in a doc, comment, or name
- a join copied from a golden's query that you have not probed as a real relationship

The golden is a hypothesis source. The data is still the verifier.

## Step 2: Probe receipts

Every structural claim needs a query you ran: join key, primary key, filter
value existence, snapshot assumption, value space, cardinality.

```malloy
run: t -> { aggregate: row_count is count(), distinct_keys is count(col) }
run: b -> { group_by: k; aggregate: n is count(); order_by: n desc; limit: 5 }
run: t -> { group_by: col; aggregate: n is count(); order_by: n desc; limit: 5 }
```

Run probes in Malloy, against the raw tables (the truth package, or a bare
`connection.table()` source), never as SQL. The second line is the join probe:
any `n` above 1 means `k` is not a key on `b`, so a `join_one` on it fans out.

A false `primary_key` compiles and silently corrupts every aggregate. Of one
pilot's 11 accepted edits, 4 of 5 wrong ones died to a single
`COUNT(*)` vs `COUNT(DISTINCT …)` probe that was never run.

**A package-skill edit still needs probes, for a different reason.** It makes no
claim about a join key, so there is nothing structural to verify, and the
`probes` array can legitimately be short. But a guide that states a fact about
the data is asserting it to every agent that reads it, with no compiler and no
query to contradict it. Probe every number and every claim about what a measure
counts before writing it into a guide, exactly as you would before writing it
into a `#(doc)`. A guide that quotes a stale figure is worse than one that
quotes none: it is confidently wrong at the moment of use.

Its reload receipt is the ordinary one. Skill files are part of the package's
content hash, so a skill-only edit moves `sourceContentSha` and an unchanged sha
means the edit did not reach what is served. After the reload, read the guide
back from the server and confirm it is your text.

Compile-check the edit before saving (scope `file` for an edit), then reload
the package. Confirm it is not serving a stale model.

This step needs a target you control: a local server, or a host that can
execute a draft. A run whose answerers queried a published model cannot be
improved in place, because publishing to score an edit is not something this
loop does. `skill:eval-loop` picks the target before the run starts, so if you
have arrived here against a published target, stop and say so rather than
publishing.

Know which copy of the file the server actually reads. Hosts commonly serve a
copy of the package rather than your working tree, so editing the model repo
and reloading recompiles the unchanged copy: the reload succeeds, nothing
changes, and a verification probe quietly tests the old model. Confirm the
edit reached what is served before you trust a probe, and keep the model repo
the source of truth that gets committed. On open-source Publisher the served
copy lives under `publisher_data/<env>/<pkg>/` unless the environment is
watch-mounted; other hosts distinguish a draft from a published version.

From malloydata/publisher, MIT.

More from malloydata/publisher