debug-failed-run

Files (1)

• SKILL.md

SKILL.md (first lines)

# Debug a failed workflow run

Start from the run id and work outward. Every step below is read-only until the
final decision, so none of it costs GPU time or destroys the evidence you need.

## 0. Find the run id

```bash
npa workbench workflow list --project <alias> --limit 50 --json
```

Runs are durable S3 objects, so this works after the controller, the cluster, or
your shell is gone. If you only have an S3 prefix, every command below accepts an
`s3://` run prefix wherever it accepts a run id.

## 1. Status first — it names the failure class

```bash
npa workbench workflow status <run-id> --project <alias> --json
```

This is the single most informative command, because **for a PENDING job it
reports the pod-level reason** rather than just "pending". That distinction is
the whole triage:

- `PENDING` + `ImagePullBackOff` → registry problem, go to step 4. The job will
  never fail on its own; Kubernetes retries pulls forever.
- `PENDING` + `Unschedulable` → capacity or accelerator-name problem, step 5.
- `FAILED` → the payload ran and exited non-zero, step 3.
- `FAILED_STARTUP` → the controller itself failed identically several times
  (threshold `--startup-failure-threshold`, default 3). This is infrastructure,
  not your payload.

Use the run's selected `--project` consistently. Status, pending-manifest lookup,
live logs, and cancellation carry the resolved project and storage credentials into
their controller calls, including a cold restart of an owned local API. The
caller environment is unchanged. This does not permit adopting a controller
under a different principal or ignoring changed credential files; reconcile
those identity failures before resuming.

All currently recorded jobs can be `SUCCEEDED` between waves while the workflow
lifecycle remains `RUNNING`. This is a successful observation of incomplete
workflow evidence, not a failed query: inspect `workflow_lifecycle` for the
original durable timestamp, `completion_recorded: false`, and
`driver_liveness: unknown`. Fresh job polls do not prove driver liveness or create
progress heartbeats. Keep the original driver running and let `--watch` cross the
handoff; do not recommend resume solely from this state. Actual query failures
still exit nonzero, and terminal workflow/artifact proof remains required.

The job aggregate and task rows are separate queue snapshots. A recognized
nonterminal job can coexist with a successful task or durable stage record while
the workflow remains incomplete. Compare each stage's `raw_job_scheduler_state`
and `raw_task_scheduler_state`; keep the original driver and watch running through
this transition. Missing/unknown/malformed job evidence and query/authentication
failures still stop verification. Contradictory terminal job/task outcomes remain
`UNKNOWN` with explicit stage conflicts; a failed parallel job may still have
successful individual tasks.

At finalization, the interpreter manifest's `completed` marker is normalized to
`SUCCEEDED`. Inspect `workflow_lifecycle.manifest_evidence` for its unchanged raw
status, timestamp, and authoritative source. This alias applies only to the
manifest; runtime ledgers require `succeeded`. Keep latest-attempt conflicts,
live-query failures, and final-artifact validation as separate checks.

Runtime-supervised runs also expose `supervisor.classification` and
`supervisor.recovery` in JSON status. `actionable_configuration` means automatic
retry has stopped and only the exact recorded attempt was cancelled;
`transient_infrastructure` records adoption or a new attempt under the same run
ID until the finite `--max-infrastructure-recoveries` policy is exhausted;
`payload` uses only explicit `--retries`; `unknown` blocks relaunch to prevent
duplicates. `INFRASTRUCTURE_RECOVERY_EXHAUSTED` is terminal durable evidence, not
permission to increase payload retries implicitly.

A credential-exec RPC stream closure during controller file sync may leave a
Pending reservation without a submitted payload. The runtime records this as a
partial launch, verifies exact cancellation and actual terminal state, checks
all declared outputs remain absent, and refreshes the shared SDK preflight
before using the existing infrastructure recovery allowance. Inspect both the
failed parent and its exact reserved successor; do not count the failed row as
running work or launch another copy. Missing/ambiguous rows, denied access,
changed identities, output uncertainty, and unverified cancellation still block
recovery. Fresh preflight denials retain their sanitized actionable reason.

Add `--watch --interval 10` to follow a run to a terminal state. Use `--cached`
only when the live controller is unreachable: its output is explicitly marked
CACHED and is **not automation-trustworthy** — never gate a decision on it.

## 2. Enumerate what the run actually produced

```bash
npa workbench workflow artifacts <run-id> --project <alias> --json
npa workbench workflow artifacts <run-id> --stage <stage-name> --json
```

Compare produced artifacts against the stages the spec declares. The first stage
with no outputs is where to look, and it is often earlier than the stage that
reported the error. A stage that "succeeded" with a zero-byte or missing declared
output is a real failure that the exit code missed.

## 3. Read the failing stage's logs

```bash
npa workbench workflow logs <run-id> --stage <stage-name>
npa workbench workflow logs <run-id> --stage <stage-name> --follow   # live job
npa workbench workflow logs <run-id> --stage <stage-name> --cached   # S3 only
```

`--follow` tails live SkyPilot logs and needs the controller; `--cached` reads
persisted object-storage evidence only and never queries the controller, which is
what you want for a post-mortem on a torn-down run. `--json` emits the log-source
contract so you can see *which* source answered — worth checking before you
conclude "there are no logs", because an empty live tail and an unread S3 object
look identical otherwise.

## 4. Registry and image problems (the most common stall)

A `403` from the registry does not fail the job, it stalls it. Listing a
repository's tags is a *different permission* from pulling it, so a `200` on
`/v2/<repo>/tags/list` proves nothing.

From nebius/nebius-physical-ai, Apache-2.0.

More from nebius/nebius-physical-ai