bench-maintain

Files (1)

• SKILL.md

SKILL.md (first lines)

# nu-validator Bench Maintenance Skill

The job is not to reduce a backlog but to **defend invariants**
across upstream updates and markuplint changes. Per-fixture verdict work is still `bench-triage`;
environment recovery is still `bench-setup`; issue-body sync is still
`bench-xref`. This skill is the loop around them.

## Invariants to defend

After every full refresh, all of these must hold before committing:

| Invariant | Check |
| --- | --- |
| No unprocessed coverage gaps | `snapshots/diff/nu-only.json` has `"entries": []` |
| No untracked exclusions | every `excluded-ids.json` reason cites a spec URL and, for `deferred-*`, an OPEN tracking Issue |
| No stale issue mappings | `yarn bench:xref --audit` reports all mapped issues OPEN |
| nu ran clean | `snapshots/diff/nu-failures.json` is empty |

A new `nu-only` entry after a refresh is never noise to sit on — it is
one of exactly three things (see decision tree below), and each has an
owner action.

## Two upstream update axes

nu-validator updates reach the bench on two independent axes; know
which one moved before interpreting a diff (`tests/external/snapshots/meta.json`
records both — compare against the committed previous version):

- **Fixture corpus** — the `tests/external/validator` submodule
  (`submoduleSha` in meta.json). Moves only when you run
  `git submodule update --remote tests/external/validator`.
- **nu engine** — the Docker image. `bench/docker.ts` pulls
  `ghcr.io/validator/validator:latest` and resolves it to an immutable
  digest (`nuValidatorImage` in meta.json) at each
  `yarn bench:update` / `--target nu` run. The engine therefore drifts
  implicitly whenever upstream publishes, even with an unchanged
  submodule.

Because the engine follows `:latest`, a "routine" full refresh can
change verdicts by itself. That is expected — the digest in meta.json
is the audit trail.

## nu update follow-up procedure

1. Bump the fixture corpus (skip to step 2 for an engine-only refresh):

   ```
   git submodule update --remote tests/external/validator
   ```

2. Full refresh (Docker required; pulls the current engine):

   ```
   yarn bench:update
   ```

3. Diff `tests/external/snapshots/meta.json` against the committed version:
   which axis moved (`submoduleSha`, `nuValidatorImage`), and do the
   totals (`totalFilesNu`, `totalNuMessages`) shift plausibly with it?

4. Classify every new `nu-only.json` entry — three causes, three
   actions:

   | Cause | How to recognize | Action |
   | --- | --- | --- |
   | New upstream fixture or new nu check | Path is new to the corpus, or the message id is new on an old fixture | Run `bench-triage` on it (spec read → rule fix, Issue, or exclusion). This is the normal intake path for new coverage. |
   | Stale per-`id` exclusion | Fixture was `nu-over`; its `excluded-ids.json` entry no longer matches because nu changed the message wording (the `nv-<hex12>` hash is derived from the message) | Re-read the entry's spec citation; if the conclusion still holds, re-pin the new id (keep the reason, update `addedAt`). If nu's new wording changes the substance, re-triage from scratch. |
   | markuplint regression | Fixture was `match-error`; ml side no longer reports | Treat as a bug in the responsible rule — find the causing commit, fix or revert. Do not exclude. |

5. Pattern-based exclusions fail in the opposite direction — silently.
   A wording change makes a `patterns[]` entry stop matching without
   any `nu-only` reappearance (the fixture just flips verdict). Check
   the `nu-over` headcount against the expected counts recorded in
   pattern `reason` fields, and re-check any `deferred-*` counts in
   the `bench-triage` "Deferred specs" table.

6. Verify all invariants (table above), then sync issue bodies:

   ```
   yarn bench:xref --audit
   yarn bench:xref --all --dry-run --write
   ```

   `--write` edits GitHub Issue bodies. Show the user the previewed
   new bodies and get their go-ahead, then apply (`--audit` rejects
   `--dry-run` / `--write`, so it stays a separate command):

   ```
   yarn bench:xref --all --write
   ```

7. Commit `snapshots/diff/*`, `snapshots/meta.json`,
   `snapshots/excluded-ids.json`, the submodule pointer, and any `issue-xref.config.ts` edits as a
   `test:` commit describing what moved (submodule range and/or image
   digest). Raw snapshot trees stay gitignored.

Before acting on any single surprising fixture, pin it with
`--concurrency 1` (see the concurrency caveat in `bench-triage`) —
parallel-run flicker is not a signal.

## markuplint-side regression check

After a rule / spec / parser change that could affect bench-covered
behavior (no Docker needed):

```
yarn bench:update:ml
yarn bench:compare
```

- `nu-only.json` must stay empty. A new entry here is always a
  regression caused by the change under test — there is no upstream
  axis involved.
- Diff the `ml-only` path set against the committed snapshot. Growth
  is acceptable only when each added path is explained by the change
  (spot-check per the "ml-only readings" note in `bench-triage`);
  shrinkage means detection was lost somewhere — explain every
  disappeared path before committing.

From markuplint/markuplint, MIT.

More from markuplint/markuplint