bench-xref

Files (1)

• SKILL.md

SKILL.md (first lines)

# nu-validator Bench Cross-Reference Sync Skill

Configured GitHub issues carry a
`<!-- bench-xref:begin v1 --> ... <!-- bench-xref:end -->` block at
the end of their body. The block lists every fixture matching the
issue's scope (filter regex) and the current verdict tally. Block
content is generated from `snapshots/diff/*` and is idempotent —
re-running only updates when the underlying numbers change.

Mappings live in `tests/external/bench/issue-xref.config.ts`.

## Commands

Flags (`--issue`, `--filter`, `--all`, `--write`, `--dry-run`,
`--audit`, `--json`) are parsed in
`tests/external/bench/xref-issue.ts` — read it when unsure. Without
`--write`, output goes to stdout and no network call is made.

Prerequisite: `gh` installed and authenticated.

Routine after a `yarn bench:update`:

```
yarn bench:xref --all --dry-run --write   # review intended diffs
yarn bench:xref --all --write             # apply
```

`composeBody` is idempotent and collapses multiple marker pairs
into one block.

The block does not adjudicate who is right between markuplint and
nu — that is a spec read per `bench-triage`.

## Updating `issue-xref.config.ts`

| Event | Action |
| --- | --- |
| New Issue this bench can verify | Add a `primary` mapping with a focused `filter`. Confirm with `yarn bench:xref --issue <N>`. |
| New Issue with no matching fixture | Add a `secondary` mapping with a `reason`. Internal-bug Issues go here too. |
| Existing Issue closes | Remove the entry. The umbrella block auto-derives from remaining primary mappings. |
| Issue scope shifts | Adjust `filter`. Rerun `yarn bench:xref --issue <N>`. |
| Body claim becomes inaccurate after bench reruns | Add or edit a `bodyOverride` factory. The recommended pattern (see its JSDoc in `issue-xref.config.ts`) is reading a co-located `.md` under `tests/external/bench/issue-xref/` — create that directory if it does not exist. |

## Pre-release checklist

Before cutting a markuplint release that touches any rule the
benchmark covers:

```
yarn bench:xref --audit                   # prune CLOSED issues from the config
yarn bench:update                         # regenerate coverage.json (Docker required)
yarn bench:xref --all --dry-run --write   # preview the new issue bodies
```

`--write` edits GitHub Issue bodies: show the preview to the user and get
their go-ahead, then run `yarn bench:xref --all --write`.

`--audit` is the cheapest gate; running it first prevents stale
xref blocks from being pushed to issues that closed between
releases.

## CI: `Bench xref audit` workflow

`.github/workflows/bench-xref-audit.yml` runs the audit on relevant
PRs and a weekly cron; it exits non-zero on any CLOSED entry,
blocking PR merge until the mapping is removed.

## Marker version

The `v1` literal is hard-coded in the `composeBody` regex. To break
format compat: bump `BEGIN_MARKER` / `END_MARKER` in
`xref-issue.ts`, temporarily widen the regex to accept both
versions, run `yarn bench:xref --all --write` once, then drop the
old version from the regex.

From markuplint/markuplint, MIT.

More from markuplint/markuplint