cdf-autonomous-vehicle-dataset-generation

Files (6)

• BENCHMARK.md
• SKILL.md
• evals.json · evals/
• airflow-direct-api.md · references/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# Autonomous Vehicle Dataset Generation

Find driving clips similar to an example video, or generate rain, snow and
daylight variations. Prepare a payload (a JSON file of run settings), run
`autonomous_vehicle_dataset_generation_dag_k8s`, and inspect the results.
The workflow uses Cosmos-Curator, Cosmos Dataset Search (CDS) and Cosmos3-Nano.
It starts and stops its own services.

For augmenting existing clips without retrieval, use
[cdf-video-data-augmentation](../cdf-video-data-augmentation/SKILL.md).
For event or anomaly videos from a seed image, use
[cdf-event-video-generation](../cdf-event-video-generation/SKILL.md).
For clip-augmentation requests where the user says no search or retrieval is
needed, stop and reroute to video data augmentation rather than building an AV
search payload or working around the search requirement.

Use a checkout matching the deployed workflow. Keep the repository documentation
and payloads available when this skill is copied into an agent sandbox.

## Component details

This workflow skill and its references are the primary guide for configuring and running the
workflow. Consult a component skill only when the requested configuration or diagnosis requires
component-specific details not covered here, and do not invent keys or values.

| Component | Covers | Component skill |
|---|---|---|
| CDF Curation and Retrieval | Cosmos-Curator clip splitting, embedding and curation; Cosmos Dataset Search collection ingest and similarity search | [`cdf-curation-and-retrieval` skill][curation-skill] |
| CDF Augmentation | Cosmos3-Nano variant generation with RGB or WSM controls, and augmentation errors | [`cdf-augmentation` skill][augmentation-skill] |

[curation-skill]: https://github.com/NVIDIA/cdf-curation-and-retrieval/tree/b4d553d00cc00c1293ad997fa8693af809489c71/skills/cdf-curation-and-retrieval
[augmentation-skill]: https://github.com/NVIDIA/cdf-augmentation/tree/73c861c43a4952b6060c87265bf39f4abc48199a/skills/cdf-augmentation

If the repository is inaccessible, continue tasks supported by this workflow's guidance. Identify
the unavailable reference and the unresolved component-specific detail only when that missing
guidance prevents completing the requested work; do not invent component settings or treat
component access as a prerequisite for ordinary workflow use.

## Examples

Choose a starter from the current [Payload Guide](../../docs/autonomous-vehicle-dataset-generation/payload-guide.md):

| Request | Starter payload |
| --- | --- |
| Find one similar driving clip | [Retrieval](../../payloads/dev/autonomous_vehicle_dataset_generation_dag/cdf-avdg-real-retrieval.json) |
| Generate one rainy variation | [Rain](../../payloads/dev/autonomous_vehicle_dataset_generation_dag/cdf-avdg-rain-generation.json) |
| Generate one snowy variation | [Snow](../../payloads/dev/autonomous_vehicle_dataset_generation_dag/cdf-avdg-snow-generation.json) |
| Generate one daylight variation | [Daylight](../../payloads/dev/autonomous_vehicle_dataset_generation_dag/cdf-avdg-day-generation.json) |

All starters process one source scene and request one match. Generation starters
produce one 93-frame variation at 1280x720. Rain and daylight use RGB controls,
24 frames per second and 35 inference steps. Snow uses WSM controls, 10 frames
per second and 50 inference steps.

Example request:

> Prepare the retrieval starter using storage profiles `inputs` and `outputs`.
> Keep the example paths and use collection `demo_av_retrieval_001`. Return the
> JSON and Airflow steps; do not submit.

Return the edited payload with generation disabled and the linked submission
steps. Explain that a successful run writes `search_result.json` and retrieved
clips under `results/` in its output folder.

For more examples, use:

- [Workflow requests](evals/evals.json): the `question` fields cover preparing
  retrieval, rain, snow and daylight payloads and explaining their results.
- [Generation prompt example](../../docs/autonomous-vehicle-dataset-generation/payload-guide.md#generate-a-variation):
  a snow edit for `cosmos.prompt`. The generation starters above contain the
  complete rain, snow and daylight prompts.
- [Retrieval walkthrough](../../docs/autonomous-vehicle-dataset-generation/getting-started.md#run-the-workflow)
  and [RGB-generation walkthrough](../../docs/autonomous-vehicle-dataset-generation/getting-started.md#next-step-generate-a-variation):
  input layout, run settings, Airflow submission and expected outputs.

## Instructions

### Prepare the Payload

Copy the selected starter. For S3-backed AV runs, use explicit
`s3://<bucket>/<path>` URIs for every storage path, including `input_video_path`,
`query_video` and `output_directory`. Replace all `msc://` placeholders in the
starter payload; use the actual backing bucket name, which may differ from the
MSC profile name. The current AV runtime does not support `msc://` paths throughout
the workflow.

For Azure requests, follow [Azure Storage with AV Dataset Generation](../../docs/autonomous-vehicle-dataset-generation/azure-support.md).
That guide requires deployment and code changes before running; changing URI
prefixes alone does not enable Azure support.

Set the user's paths and choose a unique collection name. Keep other starter
settings unless the user requests a documented change. Ask for missing inputs
or clarify whether the user wants retrieval or generation when needed.

- Source videos use `<scene-id>/camera_front_wide_120fov.mp4` and must be at
  least 9.3 seconds long. Keep outputs separate from inputs.
- The starters use `search_mode: "embedding"`. The query is processed separately
  from `limit`; all clips from its source are excluded. No remaining matches
  means the run fails.
- Retrieval keeps `cosmos.enabled: false`. Generation sets it to `true` and
  uses `cosmos.prompt` for the requested edit. `num_clips` selects matches;
  `num_augmentation` produces variations with the same prompt.
- For documented adjustments, read the [Payload Guide](../../docs/autonomous-vehicle-dataset-generation/payload-guide.md).
  It also describes indexing with `search_mode: "none"`. External service URLs,
  model selection and custom generation templates are not payload options.

For a payload-only request, return the edited JSON and unresolved prerequisites
without accessing the cluster or submitting a run.

For generation payload responses, explain the selected control mechanism:

- RGB: retrieved clips are resampled to `cosmos.fps` without changing playback
  speed, then used to derive edge controls. No WSM manifest is needed.
- WSM: each source scene's `wsm/wsm-manifest.json` identifies prepared controls
  matching the camera, frame rate, size and selected clip span. The workflow
  extracts the matching control-video section; it does not render controls.

### Check Prerequisites

From NVIDIA/skills, CC-BY-4.0 AND Apache-2.0.

More from NVIDIA/skills