cdf-event-video-generation

Files (15)

• BENCHMARK.md
• SKILL.md
• openai.yaml · agents/
• variable-distribution.json · assets/
• evals.json · evals/
• airflow-direct-api.md · references/
• outputs.md · references/
• payload-contract.md · references/
• setup-and-preflight.md · references/
• troubleshooting.md · references/
• payload.py · scripts/
• summarize_results.py · scripts/
• upload_images.py · scripts/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# CDF Orchestration — Event Video Generation

Run the Event Video Generation DAG end to end: seed-image input preparation, Cosmos3
image-to-video anomaly augmentation, auto-labeling (detection and tracking, captioning,
anomaly visual QA, person-attribute visual QA, person attribute search), anomaly dataset
generation, and result retrieval.

## Component details

This workflow skill and its references are the primary guide for configuring and running the
workflow. Consult a component skill only when the requested configuration or diagnosis requires
component-specific details not covered here. Routine payload authoring with documented fields,
preflight, triggering, monitoring, and result retrieval do not require component skill access.

When additional component guidance is needed, consult that component's documentation, and do not
invent keys or values. The links below are pinned to the latest `main` commit of each component
repository as of 2026-10-20; a component skill installed separately may describe a different
version.

| Component | Covers | Documentation at that commit |
|---|---|---|
| `cdf-augmentation` | Image-to-video configuration, generation prompts, model parameters, evaluators, and augmentation errors | [`cdf-augmentation` skill][augmentation-skill] |
| `cdf-auto-labeling` | Detection/tracking, captioning, visual QA, question banks, attribute-search configuration, and auto-labeling errors | [`cdf-auto-labeling` skill][auto-labeling-skill] |

[augmentation-skill]: https://github.com/NVIDIA/cdf-augmentation/tree/73c861c43a4952b6060c87265bf39f4abc48199a/skills/cdf-augmentation
[auto-labeling-skill]: https://github.com/NVIDIA/cdf-auto-labeling/tree/6c75c76c52a6c76c3866a38d30885e11123b0981/skills/cdf-auto-labeling

Use only documentation at these commits; do not substitute a moving branch or another version.

If the repository is inaccessible, continue tasks supported by this workflow's guidance. Identify
the unavailable reference and the unresolved component-specific detail only when that missing
guidance prevents completing the requested work; do not invent component settings or treat
component access as a prerequisite for ordinary workflow use.

Apply component guidance through this workflow's supported payload fields and DAG constraints.
Component configuration and the Airflow trigger payload are different contracts; component
capabilities do not automatically become supported DAG options. Keep execution through this
workflow's existing procedure. If the pinned guidance differs from the deployed component version
or DAG integration, report the mismatch before proposing the affected settings.

## DAG selection

The workflow builds one DAG per compute platform from
`airflow/dags/workflows/event_video_generation_dag/`:

| Platform | DAG ID | Manifest |
|---|---|---|
| Kubernetes | `event_video_generation_dag_k8s` | `event_video_generation_k8s_manifest.yaml` |
| OSMO | `event_video_generation_dag_osmo` | `event_video_generation_osmo_manifest.yaml` |

Both manifests are checked in, so both DAGs are registered. A DAG is registered only if its
manifest exists; a missing manifest means the DAG is absent from Airflow rather than broken. List
the DAGs Airflow actually loaded before triggering, and never name a DAG ID that is not in that
list.

**`event_video_generation_dag_osmo` supports external inference endpoints only.** Its manifest
defines OSMO batch tasks but no VLM/image-to-video/Cosmos-Evaluator service components, so an
internal-services payload cannot deploy what it needs:

- Always set top-level `external_services: true` (and `cosmos.external_services` to `true`, or
  omit so it inherits the top-level value). Any OSMO payload with `external_services: false` is
  invalid for the deployed manifest even though the shared payload schema can parse it — never
  submit one.
- Supply `cosmos.vlm_service_url`, `cosmos.llm_service_url`, and `cosmos.image2video_service_url`.
- Leave `service_lifecycle.vlm_service.enabled` / `service_lifecycle.image2video_service.enabled`
  as `false`, or omit `service_lifecycle` entirely.
- The OSMO DAG has no managed Cosmos Evaluator either — if any quality check stays enabled, supply
  an external Arbitrator URL via `cosmos.cosmos_evaluator_service_url` reachable from OSMO tasks,
  or explicitly disable both checks. Never reuse a K8s-DAG internal-mode starter payload for the OSMO DAG.
- Pools differ from the K8s DAG. Derive them from the OSMO manifest rather than assuming: in the
  currently checked-in `event_video_generation_osmo_manifest.yaml` the OSMO tasks
  (`detection_and_tracking`, `captioning`, `visual_qa`, `augmentation`) run on the `osmo_rtx6000`
  profile → pool `osmo_rtx6000_1`, while `event_and_person_attribute_search` stays on
  `k8s_cpu_task` → `default_pool`. (`osmo_h100_1` exists in `deploy/values.yaml` but this manifest
  does not reference it.) The OSMO manifest's `components.endpoints` key is absent — that is the
  structural reason internal mode cannot work here.

**Check `OSMO_TOKEN`, `OSMO_URL` and `OSMO_POOL` before triggering an OSMO run — and only
then.** They are optional at deploy time, so a perfectly healthy controller can be running with
them empty: `make setup` does not validate them and the chart renders empty values, so the DAG
registers and triggers normally and only fails once a task tries to reach OSMO. Do not raise this
during environment setup or for Kubernetes runs; it is a precondition of the OSMO path
specifically. Verify against the cluster (authoritative — local deployment configuration may have
changed since the last deploy):

```bash
(set -o pipefail
kubectl get secret -n sdg-workflow osmo-token-secret -o json | python3 -c '
import base64, json, sys
data = json.load(sys.stdin).get("data", {})
for key, name in (("osmo_token", "OSMO_TOKEN"), ("osmo_url", "OSMO_URL"), ("osmo_pool", "OSMO_POOL")):
    print(f"{name}: " + ("set" if base64.b64decode(data.get(key, ""), validate=True) else "unset"))
')
# A kubectl or decoding error is a failed check; never print the values or their lengths.
```

If any is unset, stop before triggering. Tell the user which settings must be configured and the
controller redeployed so the secret is recreated and the pods pick them up, and route that
configuration to the environment-setup skill. Do not set, write, or handle these values yourself.

There is a **single end-to-end pipeline per platform** — there are no generation-only or
labeling-only DAG variants. If a user asks for video generation without auto-labeling, tell them
the checked-in DAGs do not offer that flow rather than inventing a DAG ID.

## Manual payload entry in the Airflow UI

If the user wants to enter their own payload directly in the Airflow UI rather than have you
construct and trigger one, your job is limited to getting them to the UI: confirm controller
readiness, ensure `make port-forward` is running (see
[airflow-direct-api.md](references/airflow-direct-api.md#port-forward-ui-access)), and report the
reachable URL. Do not render a payload, run preflight, or trigger a run yourself in this case —
the user is doing that from the UI. Resume monitoring (step 6 below) once they tell you a run has
been triggered; you can find it via the Airflow API without needing the payload they used.

## Scope

**Before building any payload, collect all of the following from the user.** Never invent an
endpoint URL or bucket path — use only values the user supplied.

From NVIDIA/skills, CC-BY-4.0 AND Apache-2.0.

More from NVIDIA/skills