bench-setup

Files (1)

• SKILL.md

SKILL.md (first lines)

# nu-validator Bench Setup Skill

Get the bench runnable on this machine, or recover when it isn't.
Once this skill's checks pass, drop into `bench-triage` for actual
coverage work.

## Prerequisites

- Docker Desktop / Engine (macOS / Linux; WSL2 on Windows). The
  bench does not install Docker for you.
- Node 22+ (already required by the monorepo).
- The validator submodule checked out:
  ```
  git submodule update --init tests/external/validator
  ```
  This path is deliberately narrow — it is the only submodule the bench needs. A branch may carry
  others for unrelated suites (the v6 line vendors `html5lib-tests` for the Rust parser), so do not
  widen it to a bare `git submodule update --init` and assume the result is bench setup.

## Fresh-clone bootstrap

Raw snapshot trees are gitignored. Populate them once:

```
git submodule update --init tests/external/validator
yarn bench:update
```

This pulls the nu-validator Docker image
(`ghcr.io/validator/validator`), runs both tools across all
fixtures, and writes the snapshot trees, diffs, and generated spec.
Takes ~2 minutes at default concurrency.

A worktree can skip the Docker leg when another checkout on the same
machine already holds the raw trees for the same nu inputs. Compare
`tests/external/snapshots/meta.json` in both checkouts first: the same
`submoduleSha` and `nuValidatorImage` mean the nu tree there is the one
this worktree would produce. Then copy the gitignored nu tree and
regenerate the markuplint side against this worktree's build:

```
cp -r <other-checkout>/tests/external/snapshots/nu-validator tests/external/snapshots/
yarn bench:update:ml
```

## Routine refresh after dev sync

```
git submodule update --remote tests/external/validator
yarn bench:update
```

Review `snapshots/diff/summary.md` and `snapshots/diff/*.json` in
the PR. The raw snapshot trees stay local-only. What to do with the
resulting diff — classifying new `nu-only` entries, re-pinning stale
exclusions, syncing issue bodies — is the `bench-maintain` skill.

## Markuplint rule change (skip the Docker leg)

```
yarn bench:update:ml
```

Only re-runs the markuplint side; reuses existing nu snapshots.

## Commands reference

The `yarn bench:*` scripts are defined in the root `package.json`;
their flags (`--target`, `--filter`, `--concurrency`, etc.) are
parsed in the corresponding `tests/external/bench/*.ts` entry —
read those when unsure. The non-derivable bit: nu output is only
deterministic at `--concurrency 1`.

## Troubleshooting

| Symptom | Likely cause and fix |
| --- | --- |
| `Docker daemon not reachable` | Start Docker Desktop / the daemon. The bench does not install or start Docker. |
| `docker pull failed` | `docker login ghcr.io` if behind authenticated pulls, otherwise check connectivity. |
| Healthcheck times out | nu-validator takes 10–30 s to come up cold. If > 60 s, `docker logs ml-nu-validator`. |
| `Port 28888 already allocated` | Stop the conflicting container, or edit `DEFAULT_PORT` in `bench/docker.ts`. |
| `yarn bench:verify` / `yarn bench:compare` "no snapshots found" | Raw trees are gitignored on a fresh clone. Run `yarn bench:update`. |
| One or two snapshot files flip between runs | Expected nu parallelism flicker. Rerun with `--concurrency 1` when reproducibility matters. |
| Unexpected diffs after a submodule bump | Check `tests/external/snapshots/meta.json` for image-digest / markuplint-version changes. |

From markuplint/markuplint, MIT.

More from markuplint/markuplint