cdf-orchestration-setup

Files (11)

• BENCHMARK.md
• SKILL.md
• openai.yaml · agents/
• skillspector-baseline.yaml · config/
• evals.json · evals/
• controller-connection.md · references/
• deploy-controller.md · references/
• model-startup-timeouts.md · references/
• topologies.md · references/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# CDF Orchestration Setup

Deploy and verify the Airflow controller after CDF orchestration preflight has established that the
checkout and target environment are ready. This skill owns deployment and post-install operations;
it does not repeat prerequisite discovery or remediation.

## Safety boundary

Before any Helm install, image push, Airflow connection change, or remote mutation, summarize the
exact changes and obtain user approval. Never print kubeconfig, NGC keys, Hugging Face tokens, AWS
secrets, OSMO tokens, or Kubernetes Secret bodies.

## Verify CDF preflight first

Before `make setup`, a controller image build, or any Helm installation or redeployment, invoke and
follow `$cdf-orchestration-preflight` for the same checkout and target environment.

Require a fresh preflight report with `install_readiness: READY`. If installation readiness is
`BLOCKED` or `UNVERIFIED`, stop and return to preflight to identify and resolve the prerequisite gap.
Do not independently rerun prerequisite probes, including local-tool, Kubernetes, GPU, storage/NFS,
registry, credential, OSMO, or workflow-mode checks.

A `WARNING` on a tool version means the installed version is outside the tested configuration. It
does not block installation: tell the user which tool and version, note that it is untested, and
continue once they acknowledge it.

`run_readiness` may still be blocked before the first installation solely because the controller
is absent. That is expected and does not block deployment when `install_readiness` is `READY`.

If preflight identifies K3s, stop. This setup skill does not support or invoke K3s. Carry forward
the preflight report's repository root, kubeconfig, namespace, storage class, service mode, and
image repository and tag rather than rediscovering them.

Preserve the preflight invocation's image and `--probe-pod` arguments, including any exact
`--confirmed-push-repository` confirmation, for fresh and post-install audits. Keep bootstrap's
temporary probes through those checks; if one has expired, return to preflight to recreate it.
Follow its cleanup and single-GPU capacity guidance after the post-install audit.

## Select two independent axes

1. Select controller placement: use an existing controller, or deploy Airflow into the current
   Kubernetes cluster.
2. Select model-service placement separately: use external VLM/LLM/image-edit endpoints, or deploy
   those services in-cluster. Never infer one choice from the other.
3. Prefer external endpoints on a one-GPU node. The controller and augmentation worker may
   still run in that node's cluster.
**Compute backend: Kubernetes, plus OSMO for whichever DAGs ship an OSMO manifest.** A DAG is
registered per platform only when its manifest exists, so determine OSMO support by looking, not by
memory:

```bash
ls airflow/dags/workflows/*/configs/*osmo*manifest*.yaml
```

**OSMO workflows accept external endpoints only.** This is structural, not a policy preference:
every OSMO manifest defines batch tasks but an empty `components.endpoints`, so there is nothing
in-cluster for a service lifecycle to deploy. Any OSMO payload must therefore set
`external_services: true` and supply every inference URL the pipeline needs (plus an external
evaluator/arbitrator URL when a quality check is enabled). A payload with
`external_services: false` is invalid against a deployed OSMO manifest even though the shared
payload schema will happily parse it — never submit one, and never reuse a K8s internal-mode
starter payload for an OSMO DAG. Internal/in-cluster model services are a Kubernetes-only option.

Read [topologies.md](references/topologies.md) before changing infrastructure.

## Deploy a controller on the current cluster

When the user requests setup (not audit-only), choose the command sequence that matches the
preflight report, present the exact substituted commands, and obtain explicit approval.

For the build/push path, the image location is `AIRFLOW_IMAGE_REPOSITORY`/`AIRFLOW_IMAGE_TAG` in
`make/common.mk`, which preflight already verified. If that file still holds the placeholder
repository, return to preflight; do not substitute a path. Name the configured image in the prompt:

> "I will run the following commands in order:
> 1. `make setup` — validates secrets from `secrets.env` and generates the Helm values for install
> 2. `make build sdg-controller` — builds the Airflow image with DAGs and plugins baked in
>    (`docker/Dockerfile.airflow` copies `airflow/dags` and `airflow/plugins`), pushes it to
>    `<AIRFLOW_IMAGE_REPOSITORY>:<AIRFLOW_IMAGE_TAG>` from `make/common.mk`, and writes that
>    repository/tag into `deploy/values.yaml`
> 3. `make install sdg-controller` — installs/upgrades the Helm release using that image
>
> Installation first uninstalls the release and deletes all cronjobs, jobs, statefulsets,
> deployments, replicasets, services, and pods in the target namespace, including unrelated
> workloads. Most workload deletions are forced. Uninstall can remove Helm-managed PVCs,
> including model-cache and log claims. Preserve needed data before reinstall.
>
> Proceed?"

For an exact image verified by `--prebuilt-controller-image`:

> "I will run the following commands in order:
> 1. `make setup` — validates secrets from `secrets.env` and generates the Helm values for install
> 2. Create a temporary copy of `deploy/values.yaml` and set its Airflow repository/tag to
>    `<VERIFIED_REPOSITORY>:<VERIFIED_TAG>`; no image will be built or pushed
> 3. `make install sdg-controller VALUES_DEV=<TEMPORARY_VALUES>` — installs the verified prebuilt image
> 4. Run the required cluster and Airflow post-install checks, then remove the temporary values file
>
> Installation first uninstalls the release and deletes all cronjobs, jobs, statefulsets,
> deployments, replicasets, services, and pods in the target namespace, including unrelated
> workloads. Most workload deletions are forced. Uninstall can remove Helm-managed PVCs,
> including model-cache and log claims. Preserve needed data before reinstall.
>
> Proceed?"

DAGs and plugins are **baked into the Airflow image**, not synced from S3 at runtime — there is no
dag-synchronizer and no `make sync-dag` target. On the build/push path, any time
DAG/plugin/runtime-dependency source changes, or the target image tag has never been published,
`make build sdg-controller` must run before `make install sdg-controller`. A selected prebuilt image
is immutable input to this flow: never rebuild or push it. If its installed DAG set is empty or
stale, stop and report that the supplied image is unsuitable. Confirm DAGs actually loaded via the
Airflow API (`GET /api/v2/dags`, `total_entries`) rather than trusting Helm success alone.

There are exactly three controller install-related targets: `make setup`,
`make build sdg-controller`, and `make install sdg-controller`. There is no bare `make install`.

When preflight verified an exact `--prebuilt-controller-image`, do not rebuild or push it. Run
`make setup`, create a temporary copy of `deploy/values.yaml` with
`airflow.defaultAirflowRepository` and `airflow.defaultAirflowTag` set to the verified reference,

From NVIDIA/skills, CC-BY-4.0 AND Apache-2.0.

More from NVIDIA/skills