cdf-orchestration-preflight

Files (15)

• BENCHMARK.md
• SKILL.md
• openai.yaml · agents/
• readiness-profiles.json · config/
• skillspector-baseline.yaml · config/
• evals.json · evals/
• bootstrap.md · references/
• nfs-remediation.md · references/
• readiness-model.md · references/
• preflight.py · scripts/
• storage_check.py · scripts/
• skill-card.md
• skill.oms.sig
• test_preflight.py · tests/
• test_storage_check.py · tests/

SKILL.md (first lines)

# Cosmos Data Factory Orchestration Preflight

Determine what the current environment can install and which packaged DAGs it can run. Report
partial readiness precisely; do not collapse a mixed environment into a single pass/fail result.

## Safety and scope

Start with a read-only audit. Never print kubeconfig content, API keys, tokens, AWS credentials,
Kubernetes Secret bodies, or raw command output that may contain them. Do not silently use
`~/.kube/config`: use the supplied `KUBECONFIG` path for an existing cluster. For a new cluster,
bootstrap creates the kubeconfig.

This skill may offer prerequisite remediation after the audit. Before changing a host, cluster, or
tracked configuration, show the exact target and commands and obtain approval. Stop after
prerequisites are ready: do not run `make setup`, build or push the Airflow image, install the
controller Helm release, or trigger a DAG. Those are deployment or workflow tasks.

## Set up missing prerequisites

For a fresh Linux environment, read [bootstrap.md](references/bootstrap.md). It covers local
tools, the Python environment, container runtime, Kubernetes, GPU/storage setup and secrets.
Reuse working prerequisites and follow the user's execution preference. Installing the Helm
client is prerequisite setup; deploying the controller belongs to the setup skill.

## Run the audit

Run the bundled script from the repository root:

```bash
python3 skills/cdf-orchestration-preflight/scripts/preflight.py audit \
  --repo-root "$PWD" \
  --kubeconfig "$KUBECONFIG"
```

Use `--format json` when a machine-readable report will help. The script is read-only. If
`KUBECONFIG` is unset, ask for its path when a cluster exists; otherwise follow bootstrap.
Omit `--kubeconfig` to run local checks before a cluster exists. `AIRFLOW_URL` and
`AIRFLOW_TOKEN` may be exported for authenticated post-install checks; never put the token on the
command line.

When the user supplies an already-published controller image, pass its exact registry-qualified
reference with `--prebuilt-controller-image REGISTRY/REPOSITORY:TAG`. This switches installation
readiness from build/push prerequisites to a read-only `docker manifest inspect` check. For a
private registry, authenticate Docker before the audit without exposing credentials; the audit
does not log in, pull the image, or persist registry credentials. A missing or inaccessible
manifest leaves `install_readiness` `UNVERIFIED`.

For a local build, the image location comes from `AIRFLOW_IMAGE_REPOSITORY` and `AIRFLOW_IMAGE_TAG`
in `make/common.mk`. The checked-in repository is the placeholder
`nvcr.io/<your-org>/<your-team>/sdg-workflow-airflow`, which blocks the audit and
`make build sdg-controller` until it is replaced. Follow the
[getting-started guide](../../docs/getting-started.md#start-airflow-and-run-workflows): ask for an
NGC org/team path that `NGC_API_KEY` can push and a unique tag, then, with approval, set both
`?=` values in `make/common.mk`. Tell the user not to commit that edit. Exported variables of the
same names override the file for one shell; prefer the file edit because it persists across shells
and agents. After the user confirms push permission with the build credentials, pass
`--confirmed-push-repository` with the configured repository. This records user confirmation for
that exact repository, not an audit-performed push test. Without confirmation, access remains
`UNVERIFIED`. The setup skill's build step reads the same values.

For a non-loopback kubeconfig API endpoint, the audit runs a bounded `/busybox/nc` TCP probe from
an existing ready NVIDIA GPU Operator toolkit pod. A successful connection is `READY`; a
conclusive connection failure is `BLOCKED`. A loopback endpoint is always `BLOCKED`. A missing
toolkit pod or probe tool, denied `pods/exec`, or an inconclusive execution remains `UNVERIFIED`.
The audit does not create a diagnostic pod.

With a host-installed toolkit, pass `--probe-pod NAMESPACE/POD/CONTAINER` for an existing Running,
ready, non-hostNetwork container with Python 3. The first selected container tests API TCP access;
selected containers requesting `nvidia.com/gpu` also test loading `libnvcuvid.so.1` on their nodes.
Repeat the option to cover each GPU node without a toolkit Pod. Invalid selections, missing Python
or denied exec remain `UNVERIFIED`. Bootstrap includes a temporary probe Pod example.

For every node that advertises NVIDIA GPU capacity, the audit verifies `libnvcuvid.so.1`
through that node's running NVIDIA GPU Operator toolkit pod. This uses a constant, read-only
`kubectl exec` probe and does not create cluster resources. A missing library blocks workflow modes
whose readiness profile requires GPU video decode. A missing or non-running toolkit pod, a
non-Ready GPU node, or denied `pods/exec` access leaves those modes `UNVERIFIED`; do not report the
library as absent without a conclusive probe. Explicit GPU probe containers take precedence over
toolkit probes on their nodes; loading failures block decode-dependent modes.

Some modes also require operator-provisioned PVCs in `sdg-workflow` that the Helm chart does not
create. The Autonomous Vehicle Dataset Generation modes require `shared-ffmpeg`, the
read-only FFmpeg sidecar for Curator and its media probe pods. A missing or unbound claim blocks
those modes, and the report points to the
[FFmpeg sidecar prerequisite](../../docs/autonomous-vehicle-dataset-generation/prerequisites.md#shared-ffmpeg-installation).
Filling the claim is the operator's step: they choose the FFmpeg build and provision it
according to those requirements. Creating or filling the claim changes the cluster;
perform those actions only when the user requests setup. The workflow verifies the claim's contents when
its tasks start; never propose bundling FFmpeg or PyAV into the Airflow image as a substitute.

Interpret the report as two independent gates:

- `install_readiness` covers the workstation, repository configuration, credentials, Kubernetes,
  and storage needed to install the controller.
- `run_readiness` covers the live controller and every packaged DAG. `PARTIAL` means at least one
  DAG is ready and at least one is blocked or unverified.

Read [readiness-model.md](references/readiness-model.md) when explaining workflow or service-mode
results, reconciling live Airflow state, or investigating an unprofiled DAG.

## Check payload storage mapping

Workflow skills run this before every trigger. It finds every storage URL in a payload, whatever
the field name, and checks that the deployed `multistorageclient-configuration-secret` maps it,
using the same longest-prefix rule as DAG tasks. It is read-only and prints bucket prefixes only,
never credentials:

```bash
python3 skills/cdf-orchestration-preflight/scripts/storage_check.py --payload <payload.json>
```

`Storage mapping: ready` means every bucket resolves. Each `UNMAPPED` line names a field whose
bucket has no mapping; the run would fail at runtime. The generated configuration maps only
`AWS_S3_BUCKET` from `secrets.env`; to use another bucket, update it there, rerun `make setup` and
`make install sdg-controller`, then rerun the check. Use `--namespace` for a non-default
controller namespace.

## Resolve findings

From NVIDIA/skills, CC-BY-4.0 AND Apache-2.0.

More from NVIDIA/skills