cdf-anomalygen

Files (11)

• BENCHMARK.md
• SKILL.md
• evals.json · evals/
• agentic-mask-generation.md · references/
• fine-tuning.md · references/
• inputs.md · references/
• mask-placement.md · references/
• quality-refinement.md · references/
• verification.md · references/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# Skill: cdf-anomalygen

## Purpose

AnomalyGen fine-tunes the frozen **Cosmos** base into an **anomaly-generation** model: given a clean
image + defect mask, it paints a realistic defect into that region (synthetic data for detectors).
Pick a mode, collect its params, run the steps — **all commands from the repo root.**

## When to Use This Skill

Use for the CDF AnomalyGen pipeline over a defect dataset — any single stage (mask placement,
fine-tuning, generation, evaluation, refinement, or pseudo-labeling) or the full run.

- "Run the AnomalyGen pipeline on phone_screen"
- "Fine-tune an anomaly-generation model for PCB defects"
- "Generate 25 synthetic PCB anomaly images"
- "Place defect masks", "evaluate quality", or "pseudo-label the generated defects"

Do not use this skill for generic image generation, or for synthetic data that isn't defect images
built from the `{texture}/{defect}` clean-image + defect-mask contract.

## Prerequisites

- A **CUDA GPU** and the project's built env (native `.venv` or the Docker image).
- **Hugging Face auth** — run `hf auth login` — plus network access; the downloader fetches
  the model checkpoints.
- A **`dataset_dir`** in the Step 1 contract layout (clean images + `{defect}` masks; plus
  `defect_spec.jsonl` **only when `mask_backend=amp`** — the default `amg` never reads it, and
  nothing after Step 2 does either).

Preflight the env + checkpoints (GPU, Python, CUDA/torch, deps, HF auth, model checkpoints):

```shell
scripts/preflight_env_ckpt.sh
```

**Read which group failed before reacting** — the summary separates them, and only one is fixed by
downloading:

- *missing checkpoints* → `scripts/download_checkpoints.sh` (run `hf auth login` first).
- *env / Python / deps* → rebuild the environment (`bash scripts/env_setup.sh`, then activate it).

**Third group — model access for mask placement (checked HERE, never mid-flow).** The default
`mask_backend=amg` needs a vision model, and a run that discovers this at Step 2 has already spent
the user's time. A **VLM API is a base, a key and a model name** — and all three live in ONE place:
**`~/.anomalygen/endpoints.yaml`**, the model registry (nothing is built into the code; the template
is `scripts/agentic_mask_generation_serving/endpoints.example.yaml`). It sits under `$HOME`, not in
the repo, so it works from any directory, serves every checkout, and is writable inside the
read-only product containers. The moment inputs are locked with `mask_backend=amg` (the default), probe:

```shell
[ -f ~/.anomalygen/endpoints.yaml ] && echo "amg model: ~/.anomalygen/endpoints.yaml present" \
    || echo "amg model: no ~/.anomalygen/endpoints.yaml — NO model can resolve; copy scripts/agentic_mask_generation_serving/endpoints.example.yaml there"
```

**The model resolves in the registry → use it.** That is the whole check: `--gen-model` and
`--judge-model` are REQUIRED flags (there is no default model), and every name they carry must have
a registry entry (`model` + `base_url` + `key`) — the entry point refuses to start when one does
not; there is no environment fallback. A user's own OpenAI-compatible endpoint is the same case:
one `endpoints.yaml` entry with their documented base URL, their key, and their model name in
`--gen-model`/`--judge-model`.

**Anything less → stop and ask the user now**, before Step 1 — unless the user's request already
names the model (e.g. "use local Gemma"): that IS the pick, act on it without re-asking. Three
options — whichever it is, set `AMG_MODEL` to the chosen name; the Step 2 command passes it to
both required flags:

- **(a) supply an API** — one `endpoints.yaml` entry: a model name, a base and a key
  (e.g. `AMG_MODEL=azure/openai/gpt-6-astra`).
- **(b) let the skill start the model service** — in a container, as a background process inside it;
  on a dev host, from `scripts/agentic_mask_generation_serving/`. Needs a GPU; the weights are a default checkpoint —
  `scripts/download_checkpoints.sh gemma` fetches them and preflight verifies them (the group
  argument fetches only this model; bare, the script fetches all eleven).
  `mkdir -p ~/.anomalygen && cp scripts/agentic_mask_generation_serving/endpoints.example.yaml ~/.anomalygen/endpoints.yaml` wires it: the template's gemma entry already routes
  `AMG_MODEL=nvidia/google/gemma-4-31b-it` to `http://localhost:8902/v1`, and the run
  carries `--beam-multi 4 --beam-pairwise --exemplar-captions --exemplar-visual-prompt --coverage-note` — the local model always pairs with the in-loop beam and the three information channels.
  Bootstrap, lifecycle and the why: [`agentic-mask-generation.md`](references/agentic-mask-generation.md).
- **(c) set `mask_backend=amp`** for this run.

Never fall back silently. Whichever they pick, the run's own capability probe confirms the model can
take images and answer in JSON before Step 2 spends anything.

## Inputs

Collect the run's inputs from the user before starting. **If the `mode` or any input it requires is
missing, ask the user first — never guess a required input or silently fall back to its default.**

**Lock the `mode` first** — it fixes which steps run and which inputs are required (ask for any not
provided):

- **`full`** — train + generate, Steps **1 → 7**. Required: `name`, `dataset_dir`, `num_sdg`.
- **`finetune_only`** — train only, Steps **1 → 3**. Required: `name`, `dataset_dir`, `num_sdg`. Here
  `num_sdg` sizes only the validation set.
- **`generation_only`** (a.k.a. `inference_only`) — generate from an existing checkpoint, Steps
  **2, 4 → 7**. Required: `name`, `checkpoint`, `recipe`, `dataset_dir`, `num_sdg`. The checkpoint's
  geometry stamp must equal `recipe`'s. A legacy checkpoint (trained before the stamp existed) has
  none: pass its run-dir recipe copy and it is stamped from that recipe at load (generation only; it
  cannot be resumed).

`name` is required in every mode — an experiment label that names the recipe, run dir, logs and
the generated mask-set dirs. It is never a dataset location: where the data lives is `dataset_dir`,
always passed separately and never derived from `name`.

**Optional in every mode** — all have working defaults, so ask only if the user raises them:
`num_gpus` (`1`), `num_search_run` (`3`), `score` (`nn`), `task` (`texture_ft`),
`base_seed` (`1`), `mask_backend` (`amg`), `legacy_fixed_size` (off), `per_clean_pairing` (`off`).
Anything not listed as **Required** above is optional.

`legacy_fixed_size` switches Step 3 to the legacy fixed-input-size training mechanism. It is off by
default: every fine-tune uses the current variable-input-size mechanism unless the user **explicitly**
asks for the legacy one ("FT using the legacy fixed-size mechanism", "old fixed input size setting",
"previous release FT mode", etc.). Never infer it from the dataset, the image sizes, or KPI numbers,
and never ask about it when the user has not raised it. When set, the run takes
`job_name: anomalygen_texture_ft_legacy` — in the recipe and as `--job_name` to `set_pipeline_vars.sh`
— so it never shares a run dir with a default run of the same `name`, and Step 3 appends the legacy
block to the recipe.

`mask_backend` picks how Step 2 places masks: **`amg`** (default) — the agentic engine proposes,
refines and judge-accepts every mask, all defect types included; **`amp`** — the

From NVIDIA/skills, Apache-2.0.

More from NVIDIA/skills