cdf-augmentation

Files (13)

• BENCHMARK.md
• SKILL.md
• evals.json · evals/
• captioning-strategy-guide.md · references/
• config-decision-tree.md · references/
• configuration-schema.md · references/
• evaluator-setup-guide.md · references/
• event-video-gen.md · references/
• image-attribute-augmentation.md · references/
• pipeline-operations.md · references/
• troubleshooting.md · references/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# CDF Augmentation Pipeline Skill

Remote-API pipeline for captioning, generating, and evaluating augmented camera
data. Models are configured as HTTP endpoints; no local model weights are
included.

## Purpose

Use it to select a supported generation mode, author and validate
`PipelineConfig` YAML, configure captioning/evaluators, and run the
`cdf-augmentation:1.2.0` container. It covers Cosmos Transfer/Predict,
Cosmos3 WSM controls, image editing, image-to-video, BYOM endpoints, and
quality gates.

Do **not** use this skill for training or fine-tuning models, deploying clusters or NIM endpoints, or unrelated application/database development.

## Prerequisites

| Requirement | Detail |
|-------------|--------|
| **Docker** | `docker --version`. The image is **remote-API only** — it bundles no Cosmos/torch weights, so plain remote inference needs **no GPU and no `HF_TOKEN`**. |
| **NVIDIA GPU** (conditional) | Only when a configured local stage requires CUDA or decodes **H.264** in the augmentation container. See Limitations. |
| **Endpoint URLs** | One reachable URL per role used by the selected config. The examples name local Qwen services (`Qwen/Qwen3.6-27B-FP8` on `vlm`, `Qwen/Qwen2.5-14B-Instruct` on `llm`), but those are not guaranteed to be running. Resolve every required role independently: keep each reachable configured endpoint and ask only for the missing role URLs. |
| **API keys** (conditional) | Only for endpoints requiring authentication. Pass the environment variable named by each endpoint's `api_key_env`; never hardcode values in YAML. Local unauthenticated endpoints need none. |
| **Input media** | A video/image reachable by `multistorageclient` (`s3://`, `msc://`, `gs://`, `az://`, or HTTPS). Local paths are allowed **only when no external Cosmos evaluator is enabled**; external evaluation requires checker-readable remote URIs. |
| **Approved release commit** (execution only) | A 40-character commit hash confirmed by the user as release-owner-approved or retrieved from the verified release manifest. Never invent it or rely on an unverified environment value. |

## Inputs

Resolve each value in this precedence order: **state file → explicit prompt arguments → agent context → user prompt.** Ask the user only for what remains unresolved.

| Input | Required | Description |
|-------|----------|-------------|
| `config_path` | Yes | Path to the pipeline YAML, e.g. `configs/cookbook/video-data-augmentation/config_video_transfer_CT3_omni.yaml`. If absent, pick a starting config from *Supported Models* and confirm with the user. |
| `input_media` | Conditional | Source video/image → `data[].inputs.rgb`. Required for every mode except Cosmos Predict `inference_type: text2world`, where `inputs` may be null or `rgb` omitted. Overridable at run time via `data.0.inputs.rgb=...`. |
| `control_media` | For precomputed control | Control video → `data[].inputs.controls.<type>`. For Cosmos3 WSM transfer, set `controls.wsm`; the client uploads it rather than exposing its local path to the server. |
| `output_paths` | Yes | `data[].output.{video,caption,metadata}`; `evaluation` optional. |
| `model_name` | Yes | `augmentation.model.name` — an endpoint `id`, a role, or a known model name. Free-form string, not an enum. |
| `endpoint_urls` | Yes | One `endpoints[]` entry per role in use. |
| `api_key_env` | If auth | Env-var *name* per endpoint; the value comes from the environment. |
| `target_attributes` | No | `captioning.llm.variables` (e.g. `weather_condition`, `lighting_condition`). |
| `generation_params` | No | `augmentation.parameters` — pass-through; only set knobs are sent. |
| `seed` | No | Under `augmentation.parameters`. Before the first candidate, `null` resolves once to the generator/executor seed when available, otherwise the current Unix time. Retry `n` uses that resolved base seed plus `n`. `pipeline.retry` defaults to `1`, so evaluation permits at most 2 candidates by default. |

## BYOM model: endpoints, adapters, roles

Each `endpoints:` list entry declares `role`, `url`, wire `model`, and optional
`id`, `adapter`, `api_key_env`, and `timeout`. Roles are `vlm`, `llm`,
`image_edit`, `video_transfer`, `video_predict`, `image2video`, and `evaluator`.
`augmentation.model.name` resolves by endpoint `id`, then role, then the known
model-name mapping. Adapter contracts and full fields are in
[configuration-schema.md](references/configuration-schema.md).

## Supported Models

When the user hasn't specified a model, choose from their **input type and goal**:

| Input Type → Goal | `model.name` | Role / default adapter | Input → Output |
|-------------------|--------------|------------------------|----------------|
| Video — change scene attributes (weather, lighting, style) | `cosmos-transfer2.5` | `video_transfer` / `nim` | Video (+ controls) → Video |
| Video + precomputed WSM — follow world-state geometry/motion | A `video_transfer` endpoint id, e.g. `cosmos3-transfer-wsm` | `video_transfer` / `openai.video.async` | RGB video + WSM control + prompt → Video |
| Video + text — extend or predict continuation | `cosmos-predict` | `video_predict` / `nim` | Video+Text → Video |
| Text only — generate video from scratch | `cosmos-predict` (`inference_type: text2world`) | `video_predict` / `nim` | Text → Video |
| Image — edit specific attributes | `image-edit` | `image_edit` / `nim` (or `openai.chat.completions`, `openai.images.edits`) | Image → Image |
| Image — animate a first frame | `cosmos3-image2video` (or your Veo endpoint id) | `image2video` / `openai.video.sync` (Veo: `openai.video.async`) | Image + prompt → Video |

**Key rule:** use Cosmos Transfer for video scene changes, Cosmos Predict for
new/continued video, image edit for still edits, and image-to-video to animate a
frame. For Cosmos3 WSM, set `data[].inputs.controls.wsm`. Config selection
details are in [config-decision-tree.md](references/config-decision-tree.md).

All models run via remote HTTP through one `BaseExecutor`; there is no local `torchrun` and no `executor_type` field.

## Canonical Workflow

Follow this ordered decision tree; open the linked references only for the
selected mode's details.

### Config Correction Budget

`config_attempts` counts YAML versions, not validation or preflight calls. Keep
it across the whole workflow and never reset it.

1. Set it to `1` after initial authoring.
2. On a correctable error, if it is already `3`, **STOP** and report the error.
   Otherwise increment it once and make one correction pass containing all
   currently reported YAML fixes.
3. Validate and preflight that edited version without another increment. Repeat
   rule 2 only if a later check finds another correctable YAML error.

### STEP 0 — Mandatory Entry Gate

Execute this gate before every other step:

- **Scope gate:** if the request asks for training, fine-tuning, cluster/NIM
  deployment, or unrelated development, decline it and **STOP**.
- **Input gate:** resolve values using *Inputs*. If required values remain
  unresolved, request only those values and **STOP** until they are supplied.
- Proceed only when both gates pass and the input type and augmentation goal
  are known.

1. Select the model from *Supported Models* using that input type and goal.
2. Author the YAML in this order: `data`, `endpoints`, `captioning`,
   `augmentation`, `evaluators`, `pipeline`, then `data_processing`. Omit
   optional sections rather than creating placeholders. Set
   `config_attempts = 1` (**config attempt 1 of 3**).
3. Branch on the requested outcome:
   - **Configuration only or “do not execute”:** do not build, launch, run
     preflight, or infer. A trusted runtime is an already-running container
     whose recorded immutable image ID/digest, checkout commit, and config
     mounts satisfy step 4. If validation was explicitly requested and such a
     runtime exists, run only the validation command and apply step 5's failure
     rules. Return the config (mark it unvalidated if validation was not run)
     and **STOP**.
   - **Inference/run requested:** proceed to step 4.
4. Prepare the runtime in these verifiable sub-steps:
   a. Obtain `EXPECTED_RELEASE_REF` from the user as a release-owner-approved
      full commit or retrieve it from the verified release manifest.
   b. Verify the Git working tree is clean.

From NVIDIA/skills, Apache-2.0.

More from NVIDIA/skills