shiploop-e2e-audit

Files (113)

• SKILL.md
• CASES.md · harness/
• MOCK-REPLAY.md · harness/
• README.md · harness/
• SAMPLES-2026-09-17.md · harness/
• SAMPLES-2026-09-18.md · harness/
• WORKFLOW-REVIEW.md · harness/
• browser_driver.cjs · harness/adapters/
• tictactoe_20260917.cjs · harness/adapters/
• audit.py · harness/
• behavior_capture.py · harness/
• campaign.py · harness/
• capture.py · harness/
• check_suite.py · harness/
• dag_replay.py · harness/
• driver_transport.py · harness/
• evidence.py · harness/
• Bad.html · harness/fixtures/gas/benign_unused/
• Code.gs · harness/fixtures/gas/benign_unused/
• Good.html · harness/fixtures/gas/benign_unused/
• engine.js · harness/fixtures/gas/benign_unused/
• Code.gs · harness/fixtures/gas/css_relative_bad/
• Page.html · harness/fixtures/gas/css_relative_bad/
• piece.svg · harness/fixtures/gas/css_relative_bad/assets/
• theme.css · harness/fixtures/gas/css_relative_bad/
• Code.gs · harness/fixtures/gas/direct_relative_bad/
• Page.html · harness/fixtures/gas/direct_relative_bad/
• engine.js · harness/fixtures/gas/direct_relative_bad/
• Code.gs · harness/fixtures/gas/dynamic_loaders/
• Page.html · harness/fixtures/gas/dynamic_loaders/
• engine.js · harness/fixtures/gas/dynamic_loaders/
• Code.gs · harness/fixtures/gas/dynamic_rpc/
• Page.html · harness/fixtures/gas/dynamic_rpc/
• Code.gs · harness/fixtures/gas/dynamic_template/
• Page.html · harness/fixtures/gas/dynamic_template/
• Code.gs · harness/fixtures/gas/embedded_external/
• Page.html · harness/fixtures/gas/embedded_external/
• Code.gs · harness/fixtures/gas/embedded_relative/
• Page.html · harness/fixtures/gas/embedded_relative/
• engine.html · harness/fixtures/gas/embedded_relative/
• help.html · harness/fixtures/gas/embedded_relative/
• rules.html · harness/fixtures/gas/embedded_relative/
• Code.gs · harness/fixtures/gas/external_https/
• Page.html · harness/fixtures/gas/external_https/
• A.html · harness/fixtures/gas/include_cycle/
• B.html · harness/fixtures/gas/include_cycle/
• Code.gs · harness/fixtures/gas/include_cycle/
• Page.html · harness/fixtures/gas/include_cycle/
• Code.gs · harness/fixtures/gas/insecure_http/
• Page.html · harness/fixtures/gas/insecure_http/
• Code.gs · harness/fixtures/gas/missing_include/
• Page.html · harness/fixtures/gas/missing_include/
• PROVENANCE.md · harness/fixtures/gas/retained-products-v1/
• Code.gs · harness/fixtures/gas/retained-products-v1/checkers/
• Index.html · harness/fixtures/gas/retained-products-v1/checkers/
• engine.js · harness/fixtures/gas/retained-products-v1/checkers/
• Code.gs · harness/fixtures/gas/retained-products-v1/tic-tac-toe/
• Index.html · harness/fixtures/gas/retained-products-v1/tic-tac-toe/
• JavaScript.html · harness/fixtures/gas/retained-products-v1/tic-tac-toe/
• Page.html · harness/fixtures/gas/self_contained/
• …and 53 more

SKILL.md (first lines)

# ShipLoop E2E audit

Act as the **audit operator**. Run the existing harness, then explain what its
evidence establishes about ShipLoop. The harness launches the separate application
builder with the literal catalog `/shiploop` prompt. Do not start ShipLoop on the
audit request, complete its callbacks yourself, or coach the builder.

Every live case has a mandatory publication/freshness preflight. It compares
each selected ShipLoop and Improve package with the released package on
authoritative source `main` (its copy under `plugins/skill-craft`, the one plugin that
carries every skill, and that plugin's entry in skill-craft's own marketplace catalog), by
contents and executable modes.
Equal version labels alone are insufficient. If either newest source is unreleased,
either installation is stale, or either comparison cannot be verified, report the
retained freshness receipt and stop before the builder launches.
The evaluator must not publish, install, update, repoint, or repair packages
during a trial. Package preparation and authorized repairs happen between
retained trials through the campaign checkpoint below.

The external test harness owns the builder-process launch. Inside that builder
conversation, ShipLoop remains a skill plus scripts: the invoking model reads
each action packet, performs the work, and submits its callback. Do not turn
ShipLoop into a model launcher, replace this path with `shiploop drive`, or add
nested model processes to implement the audit's effort setting. Configure
`xhigh` on the external Grok launch.

## Model effort

Default every model launch used by this audit to **extra high (`xhigh`)**:
the audit operator, builder, review-only operator, independent reviewers, and
diagnostic model subprocesses. One exception: a reviewer that runs on a Claude
model (an independent reviewer or the review-only operator) defaults to
**`medium`**, because current Claude models review code well at that level;
raise it only for a measured quality gain. Pass the host's documented effort
setting explicitly (for Claude, `claude --effort medium` or an agent definition
that sets it); do not inherit an unspecified CLI default or reduce effort to
save time. Honor a different effort only when the operator explicitly requests it.
For Grok shell launches use `--reasoning-effort xhigh`; both `run.py run` and
`run.py suite` also default to it, including the low-level argv builder.

The loaded card cannot retroactively change the operator's current model turn.
Use the README's explicit `xhigh` shell invocation or confirm the current host's
effort setting. For delegated work, use an effort-selectable launch path; do not
assume a native Explore/reviewer tool inherits the parent's effort. If the host
cannot select the requested level, use an available explicit model launcher or
report that prerequisite instead of silently launching at a lower or unknown
effort. If effective effort is not observable, retain that limitation and do not
claim it was verified. Use a documented native equivalent only when
the requested level is unsupported, and record the actual value.

Git, Python/Node checks, `grok inspect`, and `--version` probes do not make model
calls and need no effort flag. Preserve an external verifier's exact argv; if
it invokes a model, configure its own model launch explicitly and audit that
setting rather than appending Grok flags to an arbitrary executable.

## Bind the installed package and choose scope

Arguments after `/shiploop-e2e-audit` are free-form instructions, not flags parsed
by a new audit CLI. Obtain the absolute path of the **selected, loaded**
`SKILL.md` from the host's skill context (expanding a host-provided alias first).
Bind `SKILL_ROOT` to that file's parent, not the user's project directory or a
guessed `~/.grok/skills` or plugin-cache path. If the host does not expose the
selected path, ask for it; do not search for another same-name card.

Run the bundled read-only [harness resolver](scripts/resolve_harness.py)
from the original session CWD before creating any output/product directories:

```sh
# Substitute the selected card's actual directory and resolved Python executable.
SKILL_ROOT="/absolute/directory-containing-the-loaded-SKILL.md"
PYTHON="/absolute/path/to/python3"
"$PYTHON" -B "$SKILL_ROOT/scripts/resolve_harness.py"
# Only for an explicit development override, add: --checkout "PATH"
# When supplied for live/check, also pass the resolved --git executable.
```

With no `checkout`, bind the selected package's bundled `harness/` directory.
This works for source symlinks and materialized marketplace/Hermes copies,
without a skill-craft clone. The current repo never overrides the selected
package. Never search sibling checkouts, infer a tree from its name, or read
installer receipts as automatic execution authority.

An explicit `checkout` is a development override, resolved against the starting
CWD. It must be a Git worktree root containing the source harness; invalid
explicit paths stop instead of falling back. Prefer reinstalling/relinking the
audit package when testing both a changed card and changed scripts. The override
selects harness files, not a different loaded audit card.

Print and retain the resolver JSON with the operator evidence: selected/resolved
card paths, `binding_source`, package and harness paths, harness digest and any
available source HEAD/dirty state. A copied package needs no Git metadata; a
digest identifies its actual harness bytes. Use the returned `harness` as
`HARNESS` and `path_base` for relative arguments. Do not evaluate JSON as shell
code. This is a progress record, not an approval step.

Read the bundled `HARNESS/README.md`
section **Run the audit from Grok** and the sections for the requested mode.
For live or retained-trial review, also read `WORKFLOW-REVIEW.md` and the current
`workflow-review-template.json`. These bundled files own commands and schemas.

ShipLoop and Improve are **separate dependencies**, never private copies inside
this audit package. For mock/apparatus checks, resolve an explicit `skill-root` or
the host-selected installed ShipLoop card and pass its parent as
`check_suite.py --skill-root`. In Grok, `grok inspect --json` from the starting
CWD must expose one unambiguous, user-invocable `shiploop` record and its
`source.path`. For live/check, it must also expose one unambiguous,
user-invocable `improve` record; Improve is selected only from that record, never
from a cache path or `--skill-root`. Inspect from the intended product CWD, then
pass the ShipLoop root to `run.py --skill-root` in the appropriate subcommand;
the runner verifies both selections again. An explicit live/check `skill-root`
must match selected ShipLoop; it does not repoint the host or select Improve. If
either subject is missing or ambiguous, report that prerequisite instead of
searching caches or installing a different package. A mock with an explicit
ShipLoop subject needs no Grok CLI or model access. Review of retained output
needs no installed subject.

| Requested mode | Operation |
| --- | --- |
| `mock` or no mode | Run `check_suite.py --suite mock` and retain its result; no live model calls. |

From whichguy/skill-craft, MIT.

More from whichguy/skill-craft