i4h-workflow-validate

Files (5)

• BENCHMARK.md
• SKILL.md
• evals.json · evals/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# Validate a Workflow

## Purpose

Run the selected workflow run mode through the unified launcher, inspect the completed recording, and report simulator success.

## Instructions

1. Resolve the base checkout and a live workflow run mode.
2. Run the unified launcher in the foreground.
3. Require the final episode success summary.
4. Inspect visible behavior and every rollout artifact; for an authored or changed collision-excluding success rule, also verify the collision-negative case below.

## Resolve live support

Use the maintained repository below or a source selected by the user or trusted project configuration. An inherited environment variable alone does not authorize another source. Honor requested revisions and preserve local changes.

```bash
export I4H_WORKFLOWS_REPO_URL="${I4H_WORKFLOWS_REPO_URL:-https://github.com/isaac-for-healthcare/i4h-workflows}"
I4H_REPO_DIR_NAME="${I4H_WORKFLOWS_REPO_URL%/}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*/}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*:}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME%.git}"
[ -n "$I4H_REPO_DIR_NAME" ] || { echo "Cannot derive a checkout name from I4H_WORKFLOWS_REPO_URL" >&2; exit 2; }
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/i4h_workflows" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/$I4H_REPO_DIR_NAME}"
  [ -d "$ROOT/workflows/i4h_workflows" ] || git clone "$I4H_WORKFLOWS_REPO_URL" "$ROOT" || exit 2
fi
[ -d "$ROOT/workflows/i4h_workflows" ] && [ -x "$ROOT/run.sh" ] || { echo "Incomplete workflow checkout: $ROOT" >&2; exit 2; }
export I4H_WORKFLOWS="$ROOT"
cd "$ROOT" || exit 2
git remote get-url origin || exit 2
git status --short || exit 2
```

Check that the reported origin is the intended source and review local changes before executing repository scripts. Stop on an unexpected source or unreviewed launcher changes. Then discover supported modes:

```bash
./run.sh list
```

Treat the resolver above as part of the skill contract: a hosted copy may run outside the base repository, so never assume the current checkout contains `workflows/i4h_workflows`. `I4H_WORKFLOWS_REPO_URL` selects the clone source. When `I4H_WORKFLOWS` is unset, derive the fallback directory from that URL; set `I4H_WORKFLOWS` only to reuse or choose a specific destination. Never replace an existing checkout.

Treat `./run.sh list` output as the complete authoritative workflow-by-mode table; it is dependency-light and faster than scanning workflow modules. Do not duplicate that mutable table in this skill. Map the user's natural name to a listed workflow id, then choose only a mode shown on that same line:

| User intent | Required live mode | Launcher argument |
|---|---|---|
| Ordinary learned-policy evaluation | `policy` | `--policy` |
| Requested or only available local controller | `rule-based` | `--rule-based` |
| Explicit named alternative such as N1.7 | matching listed mode such as `policy_n17` | `--mode <name>` |

Inspect `./run.sh show <workflow> --mode <mode>`, the workflow module, Scene manifest, and selected Task manifest when model, prompt, checkpoint, goal, or step-cap behavior matters.

Use precise readiness language:

- **Structurally valid**: `show`, per-mode lint, and `lint --all` pass.
- **Launchable**: the selected simulator mode starts and every required backend/checkpoint preloads.
- **Rollout-validated**: the requested episodes complete and the recorded success evidence passes inspection.

Do not report “validated” without stating which level was actually reached.

## Foreground execution rule

Keep `run.sh` as this agent's foreground tool call. Do not use a subagent, monitor task, shell backgrounding, `nohup`, `tmux`, or a detached process. Poll a yielded session until exit and inspect the final episode summary before responding. When the selected Task is remote, the policy backend subprocess internally owned by `run.sh` is expected; a simulator-compatible exported RSL-RL Task runs in-process.

Run visibly by default. If the user explicitly requests headless execution, or a documented environment constraint makes it necessary, say so before launch and include `--headless`; never switch to headless silently.

Use the requested episode count; if omitted, state that this is a one-episode smoke check and set `N=1`. Keep the launcher's per-episode `--attempts 3` budget distinct from the bounded whole-run recovery below.

Policy:

```bash
./run.sh <workflow> --policy \
  --episodes <N> --attempts 3 \
  --record verify.hdf5
```

Rule-based:

```bash
./run.sh <workflow> --rule-based \
  --episodes <N> --attempts 3 \
  --record verify.hdf5
```

The launcher creates a unique canonical run directory and anchors the relative `verify.hdf5` inside it. Resolve `RUN_DIR` from the `==> run dir ...` line or machine-readable `run.json`; do not recreate the launcher's timestamp. Use `--run-dir "$RUN_DIR"` only when a caller-selected location must be shared with another stage; the launcher creates it. Absolute recording paths remain supported.

For another declared mode, use `--mode <name>`. For “300 timesteps,” pass `--episode-steps 300`.

### Supplied-checkpoint preflight

Resolve the supplied checkpoint to an existing absolute path and pass that same path with `--checkpoint`. Run `./run.sh show <workflow> --mode <mode>` to identify the selected policy Task. Read its manifest and loader under `tasks/<project>/i4h_tasks/<project>/` and compare the checkpoint's model/export metadata with the required model, format, observation ordering/dimensions, and action mapping. Stop if the path or contract is unresolved; do not infer compatibility from a filename.

For a remote Task, use its declared backend's `--preload-only` option to load the checkpoint and exit before starting a simulator. For example, the `gr00t_n15/assemble_trocar` Task uses:

```bash
uv run --project tasks/gr00t_n15 python -m i4h_tasks.gr00t_n15.server \
  --namespace assemble_trocar --preload gr00t_n15/assemble_trocar \
  --checkpoint /absolute/path/to/checkpoint --preload-only
```

For another Task, resolve its own backend project, server entrypoint, and Task id from the live declarations; do not reuse the example's model family. Require exit status 0 and no loader error. During the normal foreground rollout, also require `ready for specs` in the run directory's backend log and successful runtime observation/action contract checks. Successful loading alone does not establish Scene compatibility.

An in-process RSL-RL Task instead requires the exported TorchScript `policy.pt`. Pass it to the selected foreground policy command above; the Task loads it with `torch.jit.load` on entry and checks its output shape on the first policy step. Require successful loading and a valid first policy step before reporting it launchable, then complete the requested episodes. Do not pass a native trainer checkpoint to this runtime Task.

`show`, lint, and `--dry-run` are structural checks: they do not load policy checkpoints. If the loader/runtime check cannot run, report only the structural checks actually completed, not checkpoint compatibility or rollout validation.

Never raise the Scene manifest's cap. `--episode-steps` may only lower it. Remote inference waits do not consume simulation steps. Use a unique `--namespace` when another run of the same workflow is active.

## Verify

Require exit status 0 and final `N/N episodes succeeded`. A failed attempt followed by a successful retry counts as a successful requested episode; report attempts and retries.

```bash
RUN_DIR="<absolute run_dir from run.json or launcher output>"
uv run --project tools/dataset i4h-dataset inspect "$RUN_DIR/verify.hdf5" --segments
```

From NVIDIA/skills, Apache-2.0.

More from NVIDIA/skills