submit-workflow

Files (1)

• SKILL.md

SKILL.md (first lines)

# Submit Workflow

## When To Use

Use this skill for workflow launch, YAML validation, runner scripts, and
SkyPilot submission behavior.

## Procedure

1. Read `skills/tools/skypilot-workflows/SKILL.md` for SkyPilot version and
   cleanup constraints.
2. Prefer `npa.workflow/v0.0.1` specs under
   `workflows/`. Parse / `validate-spec` locally
   before launch.
3. Use `NPA_SKYPILOT_BIN` or `npa skypilot status --bin-path`; do not assume
   `sky` from `PATH`.
4. Submit through `npa workbench workflow submit` (accepts npa.workflow specs
   and legacy SkyPilot YAML) or the shared workflow submission helper.
5. Keep cleanup best-effort and avoid tearing down a shared controller unless
   the operator explicitly requests it.

## Three-Tier Contract

- CLI: `npa workbench workflow --help` and tool-specific `workflow` commands.
- SDK: use shared workflow submission helpers rather than shelling out from
  application logic.
- YAML: author shipped workflows as `npa.workflow/v0.0.1` specs under
  `workflows/testing/`; promoted principal pipelines live in `workflows/main/`
  and partner integrations in `workflows/partners/<partner>/`. See the
  [catalog](../../../workflows/README.md) for their runbooks and validation scope.
  `npa workbench workflow submit`
  accepts those specs (plans, renders, then launches SkyPilot) and still accepts
  raw SkyPilot YAML supplied by an operator or by guarded single-task example
  directories.

## Live submit prerequisites (real cluster)

A real `npa workbench workflow submit` (not `--plan-only`) needs, on top of a
successful `npa skypilot verify --cluster <exact-context>`:

- **Secrets via `--secret-env`** (never in the YAML): `NEBIUS_TOKEN_FACTORY_KEY`
  for Token Factory stages, `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` for S3,
  `HF_TOKEN` / `NGC_API_KEY` for gated model pulls. Submit resolves each requested
  name from the explicit process environment first, then the selected project's
  configured NPA credentials; it fails locally if the value is unavailable.
- **NPA source for steps that need it.** Submit automatically stages missing
  or outdated saved source from the local editable checkout under a
  content-addressed S3 prefix and reuses verified matching source. An exported
  `NPA_SRC_S3_URI` or `NPA_E2E_NPA_SRC_S3_URI` is an explicit selection and is
  retained. For a new run after upgrading, clear an old export or use
  `--stage-src` to force restaging; leave `--no-stage-src` off for the automatic
  path. Updating NPA does not require syncing an unchanged input dataset or
  the whole bucket. Specs with `config.source_overlay: true` also apply the
  submitted NPA code inside pinned workbench images.
- **Dynamic decisions.** `--assume-decision promote_checkpoint` previews an
  accepted path in `plan-spec`, `preflight-images`, or `submit --plan-only`.
  For execution, specs declaring `metadata.executionMode: runtime`, including
  PAIDF Cosmos3, select the runtime automatically and reject assumed decisions.
  Keep `--runtime` and omit `--assume-decision` when executing that workflow;
  actual evaluator decisions control its refinement and terminal routing.
  See the [PAIDF Cosmos3 upgrade guidance](../../../workflows/guides/paidf-cosmos3.md#upgrading-an-existing-installation).
- **`--var key=value`** to override `config` (e.g. `--var bucket=<real-bucket>`;
  the reference specs default to `bucket: example-bucket`).

## Gotchas

- **Kubernetes controller launch is transactional.** Immediately before every
  Kubernetes managed-job launch, NPA probes `/readyz` on the exact selected
  context with SkyPilot's `KUBECONFIG` environment. Readiness requires three
  consecutive successes spanning 10 seconds. After any failed/uncertain launch,
  NPA reconciles exact `sky jobs queue --all --output json` evidence: adopt one
  immutable ID, retry only after authoritative absence plus a classified
  transport/API warm-up failure, or fail closed as indeterminate. Never bypass
  this with raw `sky jobs launch`, retry by name, or cancel by name.
- **An isolated SkyPilot state directory has its own stable controller user
  identity.** Reuse the same directory when resuming a run; a different isolated
  directory intentionally selects a different controller namespace. An explicit
  `SKYPILOT_USER_ID` still takes precedence.
- **A baked workflow image validates the module it actually executes.** Set
  `config.baked_npa_import` to that dotted module when `require_baked_npa` is
  enabled; otherwise the backward-compatible probe is `npa.cli.main`. This keeps
  source attestation strict without requiring narrow stage images to install the
  unrelated full CLI dependency closure.
- **Baked Kubernetes tasks get writable bootstrap caches.** NPA supplies
  pod-local `XDG_CACHE_HOME` and `UV_CACHE_DIR` defaults under `/tmp` so a
  read-only image-owned model cache cannot break SkyPilot's setup probe. Explicit
  workflow environment values still take precedence; model/checkpoint caches and
  mounted durable volumes are not redirected.
- **Explicit workload retries apply after exact resume reconciliation too.** If an
  adopted in-flight job is proven terminal, `--retries` advances through the same
  durable terminal-retry path and assigns a new attempt identity. With no explicit
  retries, the terminal outcome remains preserved and no duplicate is launched.
- **Infrastructure recovery has its own finite policy.** `--retries` remains the
  payload/terminal-wave retry count. `--max-infrastructure-recoveries` bounds
  typed capacity, quota, node-not-ready, and provider recovery per wave (default
  1; 0 disables automatic relaunch). Exhaustion is persisted and terminal; the
  two policies never silently borrow from each other.
- **Runtime supervision is durable and fail closed.** Pending pods are inspected
  by exact managed-job ID. Image/auth/reference, missing Secret/ConfigMap,
  malformed pod config, and impossible GPU shape failures stop immediately and
  cancel only that ID. Proven transient infrastructure failures may create a new
  immutable attempt under the same run ID only after immutable workflow/source/
  image identity, declared S3 output absence, preflight readiness, and exact
  cancellation are verified. Expected identities are independently recomputed
  from the current spec, source selection, and digest pins rather than copied
  from the attempt being checked. Unknown evidence blocks relaunch.
- **Async acceptance is not workload observability.** SkyPilot launch uses its
  asynchronous API mode, then the existing launch transaction reconciles the
  exact logical name to a provider job ID before runtime polling begins. Exact
  cancellation is polled to terminal; a request acknowledgement alone never
  permits relaunch.
- **Checkpoint recovery is capability-based.** Completed waves require validated
  declared outputs. Mid-stage resume requires an explicit compatible loader and
  validated application checkpoint; otherwise recovery restarts the incomplete
  wave and must not claim checkpoint resume. The same adapter contract is active
  in `npa workbench genesis train-teacher --runtime serverless` for deterministic
  Nebius Serverless Job re-attempts without a GPU supervisor VM. It does not
  enable mixed per-stage Serverless routing in `npa.workflow/v0.0.1`.
- Transaction recovery uses capped exponential jitter and a 180-second recovery

From nebius/nebius-physical-ai, Apache-2.0.

More from nebius/nebius-physical-ai