guardrail-failures

Files (1)

• SKILL.md

SKILL.md (first lines)

# Decoding A Guardrail Failure

The guardrail suite checks thousands of repository contracts. It reads the
repo and checks that surfaces which must
agree still agree. A failure almost always means one of two things — you changed
one side of a contract and not the other, or you added something the contract
requires you to register.

Run the whole suite; it is cheap enough that narrowing is rarely worth it:

```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails -q
```

Narrow once you know the file:

```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails/test_tool_catalog_argv.py -q
```

Most of these tests carry a module docstring explaining the incident that caused
them to exist. When the fix is not obvious, read it — it usually names the
production failure the check prevents.

## Triage Order

1. Read the assertion message. These tests are written to name the fix, not just
   report a mismatch.
2. Identify which side of the contract you changed. The guard is almost never
   wrong; it is reporting that a second file needs the same edit.
3. Only weaken a pinned exemption list if the exemption is genuinely obsolete.
   Several lists are explicitly designed to shrink and never grow.

## CLI, Catalog, Spec, And SDK Coherence

These fire when a tool's surfaces drift apart. This is the group a workbench
change hits most.

| Guardrail | Fix when it fails |
|---|---|
| `test_tool_catalog_argv` | A `toolRef` argv names a CLI option that does not exist, or passes a value the option cannot mean. Correct the argv in `catalog.py` — see `skills/atomic/toolref-argv-contract/SKILL.md`. |
| `test_module_toolref_argv` | A `python -m` toolRef no longer parses against its module's own argparse. Sync the module parser and the catalog argv. |
| `test_three_tier_contract` | A new CLI group has neither a `CapabilityContract` nor a seam entry, or a CLI parameter is unreachable from a spec without a stated reason. Add the contract or record the `spec_gap` category. |
| `test_spec_declared_outputs` | A spec's declared `outputs:` path is not what the tool writes. Fix the spec, or the tool's `result_uri_for()` helper. |
| `test_spec_paths_are_not_repo_relative` | A stage argv points at a path that exists in this checkout. Stages run in a pod; use an `s3://` URI or an in-image absolute path. |
| `test_shown_workflow_catalog` | A catalog entry no shipped spec reaches, or a spec outside the npa.workflow catalog. Add a reference spec or list the entry in `PUBLIC_REUSABLE_TOOLREFS`. |
| `test_workflow_schema_chaining` | A state depends on another but reads a schema that one does not produce. Align the producing and consuming stages. |
| `test_typer_command_calls` | A Typer command is called as a plain function without `@resolve_typer_defaults`. Add the decorator to the callee. |
| `test_internal_cli_entrypoint` | An internal argv no longer matches `python -m npa`. Build it with `internal_cli_argv()`. |
| `test_help_to_markdown` | The generated-docs converter mishandles wrapped Rich help. Fix `scripts/_help_to_markdown.py`. |

## Skills And Documentation

| Guardrail | Fix when it fails |
|---|---|
| `test_skills_index` | A `SKILL.md` exists without an `skills/index.yaml` entry (or the reverse), frontmatter `name` disagrees with the index, or a smoke does not run. Register the skill with a passing smoke. |
| `test_develop_skills` | A development skill names a repo path or guardrail that does not exist, or a new guardrail file is undocumented. Update the skill. |
| `test_no_dangling_workflow_references` | A doc, skill, or script names a workflow YAML that does not exist. Repoint or remove the reference. |
| `test_cli_docs_drift` | An `npa ...` command mentioned in `docs/workbench/`, `workflows/guides/`, or `skills/` does not resolve in the Typer tree. Update the doc/skill to the current command, or add a precise allowlist entry in `npa/tests/guardrails/test_cli_docs_drift.py` with the reason. |
| `test_docs_green_path` | The documented path from README to a real submit is no longer runnable end to end. Update the quickstart with the new step. |
| `test_documentation_examples` | Fix the reported Markdown link/heading, shell/Python syntax, CLI command/option, or undeclared workflow variable. Quote shell placeholders; label HTTP routes and abbreviated grammar as `text`. Keep the package README's planning example executable offline. Generated CLI references and historical command records retain separate validation scope. |
| `test_agent_workflow_operations` | The agent operations guide omitted a lifecycle stage, exposed a non-NPA subprocess, or coupled workflow operation to a model/provider flag. Restore the fixed, provider-neutral NPA command journey. |
| `test_audit_container_docs_skill` | The public container catalog disagrees with the publish inventory. Update `docs/workbench/container-image-catalog.md`. |
| `test_access_approval_skill` | Operator-owned gated access stopped being exact-artifact scoped, `Ready` became a legal or redistribution claim, or child solutions began repeating the same run-scoped use question. Restore the narrow access and inheritance contract in `skills/atomic/access-approval/SKILL.md`. |
| `test_solution_licensing_skill` | The licensing skill no longer covers an artifact boundary. Update the skill, not the test. |
| `test_third_party_eula_preflight_skill` | The EULA preflight skill is not discoverable from the operational skills that need it. Add the link. |
| `test_runtime_fetch_onboard_skill` | Runtime-fetch onboarding lost a legal boundary, safe delivery shape, byte/secret exclusion, real-workload proof, or discovery link. Restore the contract in `skills/workflows/runtime-fetch-onboard/SKILL.md` and its copyable reference; do not treat runtime fetch as permission. |
| `test_nebius_cli_compatibility` | The `nebius-cli` version drifted between packaging, `images.py`, and docs. Bump all of them together. |
| `test_skypilot_absence_contract` | The required SkyPilot version, reviewed naming-source hashes, or installed wheel no longer agree. Follow the upgrade procedure in `docs/architecture/skypilot-absence-recovery.md`; review all nine source files and refresh the contract together. |
| `test_paidf_image_tags_match_code` | Keep explicit private build tags aligned with `supported_tool_version()`, describe governed public candidates separately, and retain the fresh-cluster GPU dispatch smoke that initializes SkyPilot pull authority. |
| `test_public_runtime_registry_defaults` | A supported workload inherited a generic build/private registry, emitted an unqualified first-party image, or attached credentials to the official public release. Keep repository-owned defaults on `ghcr.io/nebius/nebius-physical-ai`; require an explicit scoped override for custom bytes. |
| `test_default_cluster_fits_quickstart` | The default cluster can no longer schedule the documented quickstart. Raise the preset or lower the spec's requests. |
| `test_workflow_readiness_records` | A readiness record is malformed, orphaned, or no longer matches its adjacent workflow SHA-256. Review the workflow-byte changes and reassess every affected claim before updating the digest; never mechanically preserve or upgrade evidence statuses. |

## Secrets, Confidentiality, And Consent

Treat every failure here as blocking. Do not add an exemption to make one pass.

| Guardrail | Fix when it fails |
|---|---|
| `test_confidentiality_scan` | The scanner, its built-in Nebius patterns, or the gitleaks wiring changed. Keep `.gitleaks.toml` and `npa/src/npa/guardrails/confidentiality.py` in sync. |
| `test_ncore_attribution` / `test_ncore_source_attribution` | NCore's exact CPython notice, immutable dual-archive proof, Git path/mode boundary, diff line mapping, or source disposition drifted. Restore the exact provenance-bound contract; never add a general exemption or hide raw findings. |
| `test_security_source` | Source scanner output, inventory, ignore handling or stable finding identity is incomplete. Restore fail-closed scanning; run the real scanner workload in `docs/security/merge-security-gate.md`. |
| `test_security_gate` | A regression comparison, snapshot boundary or dependency report check failed. Preserve duplicate detection and reject incomplete reports or source paths outside the snapshot. |
| `test_dockerfile_security_inventory` | Preserve literal exact Python pins from shell and JSON RUN instructions, separate conflicting versions across stages, reject malformed dependency-bearing instructions, and require every declared package in Trivy output. Run the real Dockerfile regression fixtures in `scripts/security_regression_checks.py`; do not replace missing coverage with an empty inventory. |
| `test_robocasa_security_inventory` | Restore `imageio-ffmpeg==0.6.0` with only sdist SHA-256 `e2556bed8e005564a9f925bb7afa4002d82770d6b08825078b7697ab88ba1755` in RoboCasa's input and lock; retain `--no-binary=imageio-ffmpeg` at install and lock generation, normalize with `pin_imageio_source.py`, and preserve the `fsspec>=2026.6.0` and `multidict>=6.9.1` floors. Reject wheel hashes and unknown direct URLs rather than widening inventory trust. |
| `test_security_install` | Scanner bootstrap no longer retries transient download failures or rejects permanent HTTP/checksum failures before extraction. Exercise real curl against a local HTTP fixture and preserve the pinned checksum. A hosted 504 during scanner installation is a transport failure; rerun its failed jobs after recovery, without retrying or suppressing security findings. |
| `test_image_security_gate` | The required security check stopped waiting for image scans, lost PR/merge-queue coverage, duplicated automatic runs, or changed existing alert identity. Restore full PR image validation or trusted identical-tree queue evidence, fail-closed result checks, distinct concurrency groups, and reporting configuration. |
| `test_security_scan_input` | The manual image-scan input became shell syntax. Pass it through a step environment variable and quote that variable in the command. |
| `test_agent_secret_guard` | A secret path became tracked, `.gitignore` stopped covering agent/cursor secrets, or a literal secret or live IP landed in agent files. Remove it and rotate. |
| `test_agent_credential_provenance_isolation` | Non-live tests inherited agent credential provenance and could reach a provider. Keep `NPA_NEBIUS_CREDENTIAL_SOURCE` in the shared environment scrub; set explicit fixture provenance only after isolation runs. |
| `test_agent_no_hardcoded_data` | Agent or insights source embeds run names, answers, or infra endpoints. Resolve them from live tool observations instead. |
| `test_access_key_list_safety` | Docs or code request secret-bearing access-key list JSON. Use a `--format jsonpath=...` projection. |
| `test_isaac_eula_plumbing` | Isaac consent uses something other than the single run-scoped mechanism. Use the scoped `ACCEPT_EULA` plumbing. |
| `test_live_tests_never_declare_a_licence` | A live test accepts vendor terms for the operator. Gate on an operator-provided variable instead. |
| `test_operator_env_covers_gated_solutions` | A solution whose smoke hits a vendor gate has no operator-env entry. Add it. |
| `test_hygiene_guards` | Several rules: no `sky launch --down/--autodown`, GPU tests skip on explicit env flags only, examples use a registry placeholder rather than a first-party ID, monolith modules must not grow, and no silent `except Exception: pass`. The message names which one. |

## Images And Packaging

| Guardrail | Fix when it fails |
|---|---|
| `test_workbench_image_k8s_prereqs` | An image lacks what SkyPilot needs on Kubernetes (python3, rsync, sudo with NOPASSWD, or the Isaac group/PATH rules). Update the Dockerfile and the shared install script together. |
| `test_isaac_skypilot_apt_closure` | An Isaac image omits an early SkyPilot transfer package or accepts a missing or removed package. Install `curl`, `fuse`, `netcat-openbsd`, `rsync`, and `wget` in both Isaac build paths, and require installed status for every package. |
| `test_unbuilt_image_records_agree` | The four files that record whether an image is built disagree. Make them agree; do not mark an image built that is not. |
| `test_seedvr2_public_release_workflows` | SeedVR2 pre-push or post-pull byte verification lost its exact image identity, local tarball, or clean-root source SHA binding. Restore both scan invocations and keep the pre-publication gate before the push. |
| `test_trivy_policy` | `trivy.yaml` no longer matches the current nested schema. Update the config. |
| `test_public_release_workflows` | The single public GHCR channel drifted: development tags must be immutable full-SHA references, all pre-publication gates must precede the push, promotion must remain exact-digest-bound, failed-build cleanup must refuse shared digests, and release health must stay anonymous/read-only. Restore those contracts in the public publication workflows; never recreate a private candidate channel. |
| `test_public_workflow_candidate_governance` | Fetch full main history, rebuild and repin a candidate from a main commit, and retain its exact digest, build evidence and qualification scope in `public_release_manifest.json`. Keep PAIDF's submitted-adapter source overlay enabled. Update the documented blocked-workflow inventory after a qualified resolution change; never conceal it with wildcard overrides or remove quarantine to make planning pass. |
| `test_secure_image_build_skill` | The secure image procedure no longer names every irreversible publication gate or its governing licensing, build, validation, and confidentiality procedures. Restore the missing gate or delegation in `skills/atomic/secure-image-build/SKILL.md`; do not weaken the test or substitute labels for artifact evidence. |
| `test_paidf_starter_asset` | The PAIDF starter asset was bundled instead of runtime-fetched, or its contract drifted. Keep the runtime-fetch model. |

## Retired Surfaces And Example Boundaries

These exist because a retired thing keeps coming back. The fix is always to use
the current surface, never to recreate the old one.

| Guardrail | Fix when it fails |
|---|---|

From nebius/nebius-physical-ai, Apache-2.0.

More from nebius/nebius-physical-ai