cdf-auto-labeling

Files (48)

• BENCHMARK.md
• SKILL.md
• evals.json · evals/
• README.md · references/
• cookbook-authoring.md · references/
• cookbook-schema.md · references/cookbook-authoring/
• scenario-authoring.md · references/cookbook-authoring/
• event-and-person-attribute-search.md · references/
• run-reference.md · references/event-and-person-attribute-search/
• event-verification-reasoning.md · references/
• run-reference.md · references/event-verification-reasoning/
• ngc-containers.md · references/
• pipeline-migration.md · references/
• migration-playbook.md · references/pipeline-migration/
• nonlinear-video-pipeline-example.md · references/pipeline-migration/
• service-reuse-matrix.md · references/pipeline-migration/
• sidecar-contracts.md · references/pipeline-migration/
• prompt-authoring.md · references/
• domain-evidence-checklists.md · references/prompt-authoring/
• prompt-patterns.md · references/prompt-authoring/
• question-bank-authoring.md · references/prompt-authoring/
• scenario-planning.md · references/
• annotation-recommendations.md · references/scenario-planning/
• intake-and-decisions.md · references/scenario-planning/
• pipeline-composition.md · references/scenario-planning/
• captioning.md · references/stages/
• detection-and-tracking.md · references/stages/
• detection-tracking-patterns.md · references/stages/
• grounding-2d-patterns.md · references/stages/
• grounding-2d.md · references/stages/
• guardrails.md · references/stages/
• person-attribute-search.md · references/stages/
• reasoning.md · references/stages/
• referring-expressions-patterns.md · references/stages/
• referring-expressions.md · references/stages/
• super-resolution.md · references/stages/
• training-export.md · references/stages/
• visual-qa.md · references/stages/
• video-data-augmentation.md · references/
• run-reference.md · references/video-data-augmentation/
• workflow-runner-debugging.md · references/
• data-handoff.md · references/workflow-runner-debugging/
• runner-debugging-patterns.md · references/workflow-runner-debugging/
• workflow-stage-integration.md · references/
• stage-service-contract.md · references/workflow-stage-integration/
• validation-checklist.md · references/workflow-stage-integration/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# CDF Auto-Labeling

Use this skill when a user wants to kick off CDF Auto-Labeling on their
own data, domain, or use case, or when the request matches a shipped cookbook,
stage, authoring, or migration task. This is a router: sequence the specialized
references instead of duplicating their detail.

## Routing (Read First)

| Request looks like | Read |
| --- | --- |
| New user, clean checkout, first validated run, "how do I get started" | This file, then the matching reference below |
| Choose annotation targets / stage subset for a domain | [`references/scenario-planning.md`](references/scenario-planning.md) |
| Create, review, or adapt a cookbook | [`references/cookbook-authoring.md`](references/cookbook-authoring.md) |
| Write or adapt VLM/LLM prompts or question banks | [`references/prompt-authoring.md`](references/prompt-authoring.md) |
| Migrate an existing annotation repo into this one | [`references/pipeline-migration.md`](references/pipeline-migration.md) |
| Run the video data augmentation cookbook | [`references/video-data-augmentation.md`](references/video-data-augmentation.md) |
| Run or choose an EPAS / PAS cookbook | [`references/event-and-person-attribute-search.md`](references/event-and-person-attribute-search.md) |
| Run event-verification reasoning | [`references/event-verification-reasoning.md`](references/event-verification-reasoning.md) |
| Debug an already-integrated workflow | [`references/workflow-runner-debugging.md`](references/workflow-runner-debugging.md) |
| Pull prebuilt stage images from NGC, log in to `nvcr.io`, or fix a missing `cdf-*` image | [`references/ngc-containers.md`](references/ngc-containers.md) |
| Implement or review a new stage or Dockerized service | [`references/workflow-stage-integration.md`](references/workflow-stage-integration.md) |
| Configure or debug one production stage | The matching file under [`references/stages/`](references/stages/) |

Stage references: [super-resolution](references/stages/super-resolution.md),
[detection-and-tracking](references/stages/detection-and-tracking.md),
[captioning](references/stages/captioning.md),
[visual-qa](references/stages/visual-qa.md),
[reasoning](references/stages/reasoning.md),
[person-attribute-search](references/stages/person-attribute-search.md),
[grounding-2d](references/stages/grounding-2d.md),
[referring-expressions](references/stages/referring-expressions.md),
[training-export](references/stages/training-export.md).

## Instructions

1. Confirm the critical run inputs with the user before doing anything else, and
   ask a concise question whenever one is missing or ambiguous - never guess or
   silently invent a default. At minimum confirm: input data path, output path,
   VLM/LLM endpoint URLs and model names, model cache path, GPU ids, and (for
   reasoning-capable models) the `max_tokens` cap. Restate the confirmed values
   back to the user before the first execution.
2. Verify the environment: repository cloned, `make` targets available, the
   model cache path exists, the VLM/LLM endpoints are reachable, and a GPU is
   available. State any missing prerequisite as a blocker instead of assuming it.
   If the stage images are not present locally, offer to pull the prebuilt
   images from NGC with `make pull` instead of building them. Use
   [ngc-containers](references/ngc-containers.md).
3. Run a shipped example first to confirm the stack works end to end before
   customizing. Pick the closest operator pipeline - video data augmentation,
   event-and-person-attribute-search, or event-verification-reasoning - and
   run its committed cookbook. Use the matching operator reference.
4. Plan the target scenario: define modality, domain, intended consumer, and
   required annotations, and get a minimal stage subset. Use
   [scenario-planning](references/scenario-planning.md).
5. Adapt the closest shipped cookbook to the new domain rather than authoring
   from scratch. Use [cookbook-authoring](references/cookbook-authoring.md).
6. Author the domain prompts and question banks. Use
   [prompt-authoring](references/prompt-authoring.md).
7. Configure the per-stage settings for the domain (detector classes or SAM3
   prompts, endpoints, windowing, `max_tokens`). Use the relevant stage
   reference, starting with
   [detection-and-tracking](references/stages/detection-and-tracking.md).
8. Dry-run the adapted cookbook, then execute and validate the outputs. Use
   [workflow-runner-debugging](references/workflow-runner-debugging.md).

Adopting an existing external annotation or dataset-generation repository into
CDF instead of starting from a shipped cookbook is a migration task; use
[pipeline-migration](references/pipeline-migration.md) for that path.

## Examples

New user, new domain: "I cloned the repo and have my own warehouse-safety video.
How do I produce auto-labels for my domain?"

Guided path:

- Confirm env (model cache, VLM/LLM endpoints, GPU), then prove the stack on a
  shipped example before customizing:

```bash
make run SCRIPT=workflow-runner:main \
  ARGS='--cookbook-file cookbooks/video_data_augmentation/configs/pipeline_video.yaml --container-dry-run'
```

- Plan the domain ([scenario-planning](references/scenario-planning.md)) -> subset
  `detection_and_tracking -> captioning -> visual_qa -> reasoning -> training_export`
  (add `grounding_2d` for caption→boxes or `referring_expressions` for boxes→phrases;
  use [grounding-2d](references/stages/grounding-2d.md) /
  [referring-expressions](references/stages/referring-expressions.md)).
- Copy the closest cookbook to `cookbooks/warehouse_safety/configs/pipeline.yaml`
  and adapt inputs, detector classes/SAM3 prompts, prompts, and question banks.
- Dry-run the new cookbook, then run for real and validate outputs:

```bash
make run SCRIPT=workflow-runner:main \
  ARGS='--cookbook-file cookbooks/warehouse_safety/configs/pipeline.yaml --container-dry-run'
```

## Guardrails

- Do not guess or fabricate the critical inputs enumerated in step 1; if any is
  missing or ambiguous, ask the user and confirm before executing.
- Do not customize a cookbook before a shipped example runs clean; a broken base
  makes domain debugging ambiguous.
- Keep the first custom pipeline minimal - only the stages needed for the
  requested annotations - and expand later.
- Verify that every selected stage's service package and image exist in the
  current branch before promising an end-to-end run.
- Do not put secrets, tokens, or absolute home paths in committed cookbooks; use
  placeholders such as `<model-cache>` and env vars for endpoint keys.
- For reasoning-capable models (for example Gemini 3 Flash), raise `max_tokens`
  on the `visual_qa` and `reasoning` LLM substages to avoid the thinking-token
  tax; keep the default cap for non-reasoning models.
- Do not rely on non-CDF pipelines, commands, or file locations. A first run
  must be reproducible through `workflow-runner:main` inside this repo.

From NVIDIA/skills, Apache-2.0.

More from NVIDIA/skills