ad-hoc-test

Files (1)

• SKILL.md

SKILL.md (first lines)

# Ad-hoc end-to-end testing of nub

A green `cargo test` does not prove the *feature works when a user runs it*. Build the dev `nub` and run the actual subcommand against a real fixture on disk — a first-class verification method, and the implementer's half of the pre-push loop. It does not replace the test suite; durable behaviors should also become committed tests.

The highest-yield bug-finding shape is a **differential fixture**: one minimal fixture isolating ONE behavior, run against `nub` AND the reference tool it claims parity with (npm/pnpm/yarn/bun/node) on identical input. Always compare against the thing you assert parity with.

**Where it runs: the builder VM by default.** Any probe or sweep that needs no macOS-specific behavior goes to a spot VM — write the whole loop below as ONE script and dispatch it with `nub scripts/remote-build.ts --job adhoc --script <file> --detach`, then `--attach <vm-name>` (the `remote-build` skill). The script runs at the synced repo root with `NUB_BIN` naming a fresh `--profile fast` build of your tree, real addon staged; the image carries Node 26 + npm, and the script installs pnpm/bun itself when a differential needs them. Batch a sweep into one script, not one VM per fixture. The merge-base control build can ride a second dispatch with `--source <merge-base-worktree>`. Stay local only for the tight fix-and-rerun loop against a warm binary and for macOS-native behavior.

## Two directions — and the second is the one that gets skipped

**CONFIRM** is the loop below. **FALSIFY** is the sweep: before opening a PR on behavior, and again before calling a review round done, hunt across many fixtures for what your change broke *somewhere you were not looking*. Both are required; only the first is instinctive.

- **Confirming is not testing.** Fixtures built to demonstrate a fix all pass, because you chose them to pass. A sweep is built to FALSIFY: adversarial shapes, real registry packages, the full tier × module- format matrix, the layouts users actually have. If every fixture passed on the first run, you tested your intent, not your change.
- **"The gates are green and the reported case works" is the trap** — it satisfies the letter of the pre-push loop, so no alarm fires.
- **Review cannot substitute for it.** A reviewer reads code and hypothesizes; they cannot run it, so they systematically miss the SILENT WRONG ANSWER — the resolution that returns a different module with no error, the value that is quietly wrong. Those are reachable only by executing the thing and checking WHICH file answered, not merely that something did.
- **Schedule the sweep explicitly.** Answering someone else's findings always offers a next one; your own verification never gets scheduled by default.

The sweep decomposes into prongs that share nothing — adversarial boundary abuse, real installed packages, monorepo and symlink topologies, the tier × entry-kind × import-form matrix — so it parallelises across sub-agents well. Require every claim to carry its command and verbatim output, and verify each load-bearing finding yourself.

## The control decides whether the result means anything

A green result with no control is not evidence. Run each fixture three ways — plain `node`, a build of your branch's **merge-base**, and your build.

- **The control is the MERGE-BASE, never a shipped release.** `~/.nub/bin/nub` can trail `main` by dozens of unrelated commits, so a difference against it is not attributable to your change and every "pre-existing, not mine" verdict resting on it is unfounded.
  ```bash
  git worktree add /tmp/base $(git merge-base origin/main HEAD)
  # own CARGO_TARGET_DIR; build the addon before nub-cli
  ```
- **Bind the artifact to the change before trusting any run.** The N-API addon and the whole `runtime/` tree are EMBEDDED into the `nub` binary at build time, so a stale binary silently tests the old behavior and a working fix reads as broken. Rebuild in order (addon → `nub-cli`) and confirm from the build log that the crates you changed actually compiled.
- **Score with the SCORER, never by hand-summing its input.** A results file is often PRE-GATE: in the build-jail corpus, `verdict-table.json` holds raw rows while validity, scheduling, artifact-ratio and manufactured-pass all live in `aggregate.mjs`, so arithmetic over the tables silently skips every gate. Measured 2026-07-31: hand-summing gave 47 survive / 61 break where `node aggregate.mjs <root>` over the `report.json` set gives **45/63** — two `hook-installer` rows that read `DID-WORK-AND-SUCCEEDED` in BOTH arms are still breaks, because a granted shard-mate's project write satisfies an ungranted package's predicate. Run the project's own scorer over its own inputs; a number you computed yourself from an intermediate file is a different number.
- **Read the artifact you BUILT, not the first path matching its name.** Same root cause as the rule above — trusting the first plausible file instead of the final one. `--profile fast` writes `target/fast/`, `cargo test` writes `target/debug/`, and generated files live under a per-crate `build/<crate>-<hash>/out/`, so several plausible copies coexist and the stale one is often first alphabetically. Sort candidates by mtime and check the winner is newer than the build.
- **Vary exactly one thing, and check the variable is INDEPENDENT of the harness.** A control that moves two variables launders a wrong answer into a verified one; a variable that also feeds the measurement destroys attribution instead. Measured 2026-07-31: a Windows differential varied `LOCALAPPDATA`, but nub's store lives at `%LOCALAPPDATA%\nub\pm\store`, so the store moved outside every snapshot root and 6 packages read NOT-INSTALLED. Before running an experiment, ask whether the thing you are varying also feeds the harness. When a control agrees with what you expected, get suspicious rather than relieved.
- **Exit code is NOT a universal success condition — check what the script actually does first.** Some lifecycle scripts wrap `spawnSync` in `try/catch` and never propagate its status, so *every* arm exits 0 no matter what was denied (every hook installer measured — `lefthook`, `@arkweid/lefthook`, `@evilmartians/lefthook` — behaves this way). Where that happens, judge the ARTIFACT against a jail-off control instead. Exit code remains the right default; this is a real exception class.
- **A grant can MOVE a failure rather than close it.** Re-measure after every grant instead of assuming one is the whole job.
- **Cover the TIERS, not just your host Node.** The fast tier (22.15+) and compat tier (18.19–22.14) take different code paths and break differently.
- **A behavior fix landed AFTER your sweep un-verifies the sweep.** Narrowing changes feel safe but can withdraw a resolution real code depended on. Re-run at least the highest-value prong against the binary you intend to ship.
- **Read a tester's CAVEATS as closely as its findings** — the limitations section is where a report tells you which conclusions are not load-bearing yet.

## The loop

### 1. Create a fixture in a tmp dir

Minimal, isolating the ONE behavior you changed — not a whole app.

```bash
FIX=$(mktemp -d /tmp/nub-fix.XXXX)
cd "$FIX"
# write only what the behavior needs, e.g.:
cat > package.json <<'EOF'
{ "name": "fix", "scripts": { "build": "echo built" } }
EOF
# ...a lockfile, a workspace, a .npmrc, a tsconfig, an index.ts — whatever this behavior reads.
```

> **Give each fixture a UNIQUE package identity when the behavior touches install / linking /
> build-approval.** nub's global virtual store persists built package cells across runs, so a second
> fixture reusing a dependency's `name@version` can link the *first* run's already-built cell — a FALSE
> CONFIRMATION, because the run never exercised your package at all. Name the dependency uniquely per run
> (e.g. `dep$(date +%s)@9.9.9`) or wipe the store, and confirm `node_modules/<dep>/` contents are yours.

### 2. Build the dev `nub`

```bash
# from your worktree:
scripts/rust-build.sh build -p nub-cli --profile fast
NUB="$(scripts/rust-build.sh --print-target)/fast/nub"      # DERIVE it; see below
ls -l "$NUB"                                                # and eyeball the mtime
# or, if you ran `make install-dev`:  NUB=nub-dev
```

**Never hardcode `<worktree>/target/fast/nub`.** The wrapper builds into a content-hashed bucket under `~/.cache/nub/shared-target-<hash>/` AND CoW-seeds the worktree's own `target/` from a warm bucket — so `<worktree>/target/fast/nub` exists, is executable, reports a plausible `--version`, and is not what you just built. There is no error and no warning. Measured 2026-09-05: a probe script read that path for an hour while the real build sat in the bucket, and the stale copy happened to hold an earlier version of the same change, so every run looked plausible; the tell was the sweep finally disagreeing with a unit test. Derive the path from `--print-target` and print its mtime, so a stale read shows up in the output instead of being inferred later. (`cargo test` is immune — it compiles from source.)

If the change touches the runtime/transpiler (the N-API addon), build the addon too: `make addon-fast` (or `make install-dev`, which does both).

### 3. Run the subcommand against the fixture

```bash
cd "$FIX"
"$NUB" <the-subcommand-and-flags-you-changed>
echo "exit: $?"
```

### 4. Verify the INTENDED effect

State explicitly what "worked" means before running, then check that exact thing — the effect on disk, the exit code, or the diff vs the reference tool.

```bash
# filesystem effect:
ls -la node_modules/.bin/ ; cat the-file-it-should-have-written
# lockfile/config it should have produced or respected:
cat nub-lock.yaml 2>/dev/null; cat package.json
# exit code on a refusal path:
"$NUB" <unsound-invocation>; echo "exit: $?"   # expect a non-zero + a clear error
# DIFFERENTIAL: run the reference tool on the SAME fixture and diff:
pnpm <equivalent> ; # compare output / node_modules / lockfile to nub's
```

### 5. Probe variants and edge cases

```bash
"$NUB" <cmd> --flag-variant        # each flag/alias you touched
"$NUB" <cmd>                        # the no-arg / default case
(cd empty-dir && "$NUB" <cmd>)      # missing input / empty project
"$NUB" <cmd> <malformed-input>      # the failure mode — should error clearly, not panic
```

For version-banded runtime behavior, drive nub onto a specific Node: `PATH="$HOME/.nvm/versions/node/v20.19.0/bin:$PATH" "$NUB" …`. Use Docker for clean-machine / global-cache / Node-floor behavior.

### 6. Clean up

```bash
rm -rf "$FIX"
```

## Then: promote durable checks into the suite

Ad-hoc verification proves *this* change; a committed test prevents the *next* regression.

- A behavior covered by a tmp-fixture check should become a committed integration test under `crates/nub-cli/tests/*.rs` (or a documented harness under `tests/<feature>/` for multi-version/Docker loops).

From nubjs/nub, MIT.

More from nubjs/nub