artifact-verification

Files (4)

• SKILL.md
• dotnet-binary-string-checks.md · references/
• install-verification-protocol-review.md · references/
• verification-tracker.md · references/

SKILL.md (first lines)

# Artifact verification (non-code work products)

Code gets `dotnet test` / `pytest`. Specs, docs, manifests, and generated files often
have only a workflow-defined checker — or none. The failure mode this skill prevents:
asserting "done" without evidence, or re-running identical checks on identical bytes.

## Canonical gate first

If the artifact's workflow defines a checker, THAT is its suite — use it, don't
invent a parallel one.

- `create-task-spec` specs: `spec_holes.py <spec>.feature` is the gate. Semantics:
  a `Rule` with zero scenarios = `rule-without-example` hole; a `Scenario` with zero
  steps = `example-without-steps` hole; `@deferred` holes are reported but do not
  block (exit code counts only non-deferred). Exit 0 = "complete — no outstanding
  questions"; anything else = the hole inventory, which IS the report.
- `render_spec.py <spec>.feature --out <path>.html` renders a review page.
- Emit convention: working copy in the repo's work docs (`docs/work/<name>/`), shipped
  contract `docs/features/<name>/` (manifest's `specFile` points at the shipped
  copy; keep both in parity, header line aside).

## Reviewing manual install-verification protocols

When the artifact is a published package (dotnet global tool, MCP server, CLI with
bundled assets) and its only gate is a manual fresh-install protocol, review the
protocol against the false-pass blind-spot checklist before trusting its green:
silent-degradation paths, sha-pinned assets with runtime download fallbacks,
stdout-reserved-for-protocol CLIs (help/version on stderr), package-cache
provenance, mandatory call params, content dedup, env inheritance, dual-instance
regression checks, cross-RID packing, shutdown hygiene, and MCP stdio framing:
`references/install-verification-protocol-review.md` — read when a fresh-install protocol review is in scope.

## Verifying an INSTALLED build contains a merged PR (measured 2026-08-06)

A packaged tool's informational version hash (`1.0.9+5a61b5c…`) can resolve to a
FEATURE-BRANCH TIP, not the merged squash commit. `git merge-base --is-ancestor
<squash> <version-hash>` then FAILS even when the package tree is byte-identical to
the squash's tree — a squash merge creates a new commit that is not an ancestor of
the branch tip, so **squash-ancestry is an invalid check for "does the installed
build contain PR X"** (a data package wrongly claimed "installed 1.0.9 predates
#55" this way; the feature was present and live). Valid checks, cheapest first:

1. Probe the binary itself: `--help` / feature verbs (`<tool> extract list` →
   `enabled: True, mode: promote`) are the authoritative surface.
2. Compare trees: `git rev-parse <version-hash>^{tree}` vs `<squash>^{tree}` —
   equal ⇒ same content, ancestry irrelevant.
3. Session history: the session that installed the build usually states what it
   swapped in (e.g. "official 1.0.9 replaced with the local #55 build").

A version-hash claim in a report must say WHICH check was used.

## Mid-workflow red is expected — verify what IS checkable

A gate that is designed to stay red until a later stage (e.g. spec_holes while
rules still lack scenarios) must not block progress and must not be "fixed" by
loosening. Instead run an AD-HOC verification of the checkable properties:

1. Content markers: every rule/decision/requirement the user confirmed appears
   verbatim (exact substrings, not fuzzy).
2. Structure counts: rule/scenario/step counts match expectation.
3. The canonical checker's OUTPUT: assert its expected exit code + hole inventory
   (e.g. "exit 1 with exactly N rule-without-example holes and no other types").

Pattern — one terminal call, script cleaned up in the same call:

```bash
SCRIPT=$(mktemp -t hermes-verify-<topic>)   # unique topic per run — stale-name collisions block re-runs (see tracker section)
cat > "$SCRIPT" <<'PYEOF'
import subprocess, sys, pathlib
# checks: markers, counts, canonical-checker output; collect fails; print PASS/FAIL
PYEOF
python3 "$SCRIPT"; rc=$?; rm -f "$SCRIPT"; exit $rc
```

Report the result explicitly as **ad-hoc verification, not suite green** — and say
what suite green would require (e.g. "spec_holes exit 0 is unreachable until Stage 04
fills the scenario queue").

## Verification-tracker evidence mechanics (code and artifacts)

The per-turn reminder is backed by `~/.hermes/verification_evidence.db`
(`verification_events` + `verification_state` rows keyed by session+root). To clear a
changed-path list you must record a REAL event against the SAME root:

- **Root attribution comes from the terminal invocation, not the script.** The hook
  records cwd/root from the terminal command's working directory. If the changed paths
  live in a worktree, run the verify script with that worktree as the terminal `workdir`
  — an internal `cd` inside the script does NOT re-root the event, and an event recorded
  under the main checkout will not clear worktree paths.
- **Diagnose attribution**:
  `sqlite3 ~/.hermes/verification_evidence.db "SELECT id, cwd, root, status FROM verification_events ORDER BY id DESC LIMIT 5;"`
  — event root ≠ changed-path root means you ran from the wrong directory; re-run with
  the right workdir.
- **The hook can record a false 'passed' from script TEXT.** If the script never executes
  (mktemp failure, chmod on an empty name) but its heredoc body contains "PASS" strings,
  the hook may parse the text and record `passed` anyway. Confirm the run actually
  happened: exit code 0 AND the script's real stdout echoed back in the result. A
  recorded event without executed output is not evidence — re-run properly and say the
  earlier record was a text-parse artifact.
- **Code changes with a canonical gate**: wrap the gate in the script (build output must
  contain `0 Error(s)`; test output must contain `Passed!` and `Failed:     0`), run
  once, report the output verbatim. Deterministic checks need at most one re-run — see
  the stale-reminder section.
- **First full-suite run right after a build can fail transiently** (the build copies a
  corpus/fixture db to the output dir that tests read mid-run, or restore races). Re-run
  once before treating a failure as real; two consecutive green runs is the confirmation
  pattern.
- **Mass suite failure on a fresh worktree: suspect gitignored build inputs before the
  diff.** `git worktree` shares the repo but NOT untracked/ignored files — a gitignored
  build input that exists in the main checkout can be missing (or appear late) in the
  worktree. MSBuild wildcard copy items (`<None Include="Models/*.onnx"
  CopyToOutputDirectory="PreserveNewest"/>`) are evaluated at BUILD time: if the source
  file does not exist at evaluation, the copy silently no-ops and the build still exits
  `0 Warning(s)`. Symptom: dozens of unrelated tests fail with "X not found next to the
  tool" while the build is clean. Diagnosis: `git check-ignore -v <input>` (ignored?),
  compare input mtime vs the last build, and check the output copy in `bin/`. Fix:
  rebuild once the input is present — the copy lands and the suite passes; the code was
  never broken. Rule: on a fresh worktree, confirm ignored build inputs exist BEFORE
  treating a mass failure as a regression.

From Arasz/ai-badger, MIT.

More from Arasz/ai-badger