bench-triage

Files (2)

• SKILL.md
• audit-log.md

SKILL.md (first lines)

# nu-validator Bench Triage Skill

Take one nu-only fixture and drive its verdict to a confirmed state.

Every entry this skill sees is fresh intake — surfaced by a bench
refresh after an upstream nu update or a markuplint change. Deciding
*why* the entry appeared (new upstream coverage vs. stale exclusion
vs. markuplint regression) is `bench-maintain`'s decision tree; this
skill is the spec-read that settles the verdict once the entry is
confirmed as new coverage to classify.

Prerequisite: the bench must be runnable on this machine. If commands
in this skill fail with "no snapshots found" / Docker errors, run
the `bench-setup` skill first.

## Verdict definitions

| Verdict | Meaning |
| --- | --- |
| `match-error` | Both tools detected a violation. |
| `match-clean` | Neither detected a violation (and no nu errors were excluded). |
| `ml-only` | Only markuplint detected. |
| `nu-only` | Only nu-validator detected, and `excluded-ids.json` does not cover the messages. |
| `nu-over` | Only nu-validator detected, but every message is covered by `excluded-ids.json`. |

`nu-only` is what this skill drives. `ml-only` is informational and
not this skill's target — but if you need to understand it, see
"Note: ml-only readings" at the end.

A verdict tally (`ml-only=32` for a category, the count line in a
`bench:xref` block) is a pointer, not evidence. It says how many
fixtures differ, not which rule or attribute differs or whether
markuplint or nu is the one following the spec. Before a tally is
quoted in an Issue or PR body, pick one or two representative
fixtures, regenerate their snapshots (`yarn bench:update --target
markuplint`, or run them through `mlTest()`), read
`snapshots/markuplint/<path>.json` `violations[]`, and confirm the
flagged rule and attribute are the ones the claim is about. An
`ml-only` count read as "markuplint over-detects" without that step
has produced Issues whose actual violations came from unrelated
rules.

## Step 1: Pick a fixture

Slice `tests/external/snapshots/diff/nu-only.json` (`entries[]` with
`category` and `path`) by path or category. When auditing a coverage
claim instead, slice `coverage.json` by the claim's pattern and read
each entry's `verdict`.

## Step 2: Read nu-validator messages for the fixture

Read `tests/external/snapshots/nu-validator/<path>.json`
(`nuValidator.messages[]`). The raw tree is gitignored — regenerate
with `yarn bench:update --target nu` if missing. Each message has a
stable `id` (`nv-<hex12>`, optionally `-N` on collisions); that's
the key for `excluded-ids.json`.

## Step 3: Read markuplint output for the fixture

Read `tests/external/snapshots/markuplint/<path>.json`
(`markuplint.violations[]`). For a `nu-only` fixture, expect zero
violations here. If markuplint already detected something, the
verdict computation may be stale — re-run `yarn bench:compare`.

## Step 4: Read the spec

Open the raw HTML at `tests/external/validator/tests/<path>` and
identify the relevant spec paragraph. Authoritative sources:

- HTML LS — <https://html.spec.whatwg.org/multipage/>
- DOM LS — <https://dom.spec.whatwg.org/>
- URL LS — <https://url.spec.whatwg.org/>
- WAI-ARIA 1.3 — <https://www.w3.org/TR/wai-aria-1.3/>
- ARIA in HTML — <https://w3c.github.io/html-aria/>
- Microdata (HTML LS §5.7) — <https://html.spec.whatwg.org/multipage/microdata.html>

MDN is not authoritative — quote WHATWG / W3C when they disagree.
Living standards change; recent normative revisions often explain
why nu (slow) and markuplint (tracks `@markuplint/html-spec`) drift.

Quote the exact sentence verbatim into the issue / PR /
`excluded-ids.json#reason` — never a paraphrase.

## Step 5: Decide and act

For a `nu-only` fixture, the spec verdict gives a binary action.

Where a fix lands depends on the branch: the paths named throughout this skill (`packages/@markuplint/types`, `packages/@markuplint/html-parser`, …) are the `dev` line. On the v6 line the same behaviour may be implemented in the Rust workspace instead — locate the code before editing, and follow `crates/markuplint-rules/CLAUDE.md` for anything under `crates/`.

| Spec on the markup | Conclusion | Action |
| --- | --- | --- |
| **Forbidden (HTML LS / ARIA / URL LS)** | nu correct, markuplint has a coverage gap. | Add or extend a markuplint rule. Open an Issue if the work is non-trivial. After fix, `yarn bench:update:ml` — fixture should flip to `match-error`. |
| **Forbidden, but spec is outside markuplint's reference scope** (e.g. WICG draft, vendor extension) | nu is enforcing a spec that markuplint deliberately does not track. Open an Issue for future coverage AND record the messages in `excluded-ids.json` so the bench can focus on actionable HTML LS gaps. | Issue + `excluded-ids.json` pattern. Reason field must explicitly note `deferred-WICG / deferred-<spec>` so future readers can distinguish from regular nu-over. The tracking Issue # goes in the reason. |
| **Permitted by HTML LS** | nu over-detecting. | Record in `excluded-ids.json` (per-ID or pattern; see below). After edit, `yarn bench:compare` — fixture should flip to `nu-over`. |
| **Ambiguous / under discussion** | Spec issue or PR ongoing. | Put the spec-tracker URL in the `reason` of the markuplint tracking Issue's `secondary` mapping in `tests/external/bench/issue-xref.config.ts` (file or find that Issue first — a mapping is keyed by its number). `summary.md` is regenerated on every `bench:report`, so a note there is lost. Do not silently close. |

markuplint's reference scope is HTML Living Standard + WAI-ARIA +
URL Living Standard. Anything nu enforces from a WICG draft, a
vendor extension, or any other spec outside that set is treated
as deferred coverage — eligible for `excluded-ids.json` only if
an Issue tracks the future implementation.

When the spec disagrees with **both** tools (recent normative
revision neither has adopted), open one Issue per tool but pursue
only the markuplint side from this repo — nu upstream reports are
not part of this project's workflow.

### How to record nu over-detection

Follow the existing entry shapes in `excluded-ids.json` (per-`id`
`entries[]` keyed by the nu message id; message-substring
`patterns[]` keyed by `messageContains`). Every entry needs a
`reason` containing the verbatim spec quote, plus `addedAt` /
`addedBy`.

Message-substring conclusions are kept in the [audit log](audit-log.md);
check it before excluding by message.

The verdict flips to `nu-over` only when *every* active nu message

From markuplint/markuplint, MIT.

More from markuplint/markuplint