nv-segment-ct-finetune

Files (11)

• BENCHMARK.md
• SKILL.md
• evals.json · evals/
• datalist.json · fixtures/spleen_micro/
• task06-and-results.md · references/
• run_finetune.py · scripts/
• skill-card.md
• skill.oms.sig
• skill_manifest.yaml
• test_run_finetune.py · tests/
• output_schema.json · validators/

SKILL.md (first lines)

# NV-Segment-CT Finetune

## Purpose

- Used for smoke or dataset finetuning of NV-Segment-CT VISTA3D on CT NIfTI labels, including the upstream fixed-channel softmax workflow and optional MLflow tracking. Not for clinical validation.
- Wraps the upstream MONAI bundle entrypoint; do not replace it with handwritten training or inference code.
- Manifest inputs are `dataset_dir`, `datalist`, `target_anatomy`, `label_mapping`, `smoke`, `sanity`, `auto_seg`, `softmax`, `skip_formal_eval`, `mlflow_tracking_uri`, `mlflow_experiment_name`, and `mlflow_run_name`.
- Manifest outputs are `finetuned_ckpt` and schema-checked `result_json`.

## Instructions

- Run only the caller-requested preset, dataset, output directory, and compute budget. Dependency setup and remote tracking require the caller's approval. After reporting the result, stop; further training, publishing, or deployment is a separate request.
- Run `scripts/run_finetune.py`; do not patch files under `bundle/` or upstream checkouts during normal skill use.
- For standalone Bash, include the fresh-environment setup line before the wrapper; benchmark venvs start empty.
- Run the committed script in place from the repo root. Do not copy this skill to a runtime directory, and do not use `rm` or cleanup commands in generated invocations.
- If a host exposes `run_script`, use `run_script("scripts/run_finetune.py", args=[...])`; otherwise run from the repo root.
- For the shortest workflow check, use `--smoke`; for MSD Task06 Lung Tumor reproduction, use `--sanity`.
- Choose between the standard and `--softmax` workflows using the criteria below. Do not combine `--softmax` with `--auto-seg` or `--sanity`.
- Set `--mlflow-experiment-name` to enable MLflow for the training phase of either workflow. `--mlflow-tracking-uri` and `--mlflow-run-name` require an experiment name. Formal pre/post evaluation does not receive MLflow credentials.
- Read `references/task06-and-results.md` only when you need Task06 reference details, output-field definitions, or manual bundle setup notes.

## Choosing the Workflow

Use `--softmax` only when all of these conditions hold:

- The complete class set is known before training and will not vary between inference requests.
- Labels are mutually exclusive: each voxel is background or exactly one foreground class.
- Every foreground dataset label maps to an existing VISTA3D class ID, and a conventional fixed-channel output is desired.

Keep the standard workflow if point prompts must remain available, classes are selected dynamically at inference, labels can overlap, or the Task06 `--sanity` reproduction is required.

For `--label-mapping '[[1,3],[2,13]]'`, channel 0 is background, channel 1 represents dataset label 1 initialized from VISTA3D class 3, and channel 2 represents dataset label 2 initialized from VISTA3D class 13. Preserve the entries and their order when using the resulting `model_softmax.pt` with upstream `configs/inference_softmax.json`. The `nv-segment-ct` and `nv-segment-ctmr` inference skills do not currently expose that fixed-channel inference path.

## Available Scripts

| Script | Purpose | Arguments |
|---|---|---|
| `scripts/run_finetune.py` | Primary entrypoint declared by `skill_manifest.yaml`; stages configs, runs MONAI, and writes `output.json`. | `[FIXTURE_OR_DATASET] --output-dir OUT_DIR [--smoke] [--sanity] [--auto-seg] [--softmax] [--dataset-dir DIR] [--datalist JSON] [--target-anatomy TEXT] [--label-mapping JSON] [--patch-size JSON] [--mlflow-experiment-name NAME] [--mlflow-tracking-uri URI] [--mlflow-run-name NAME]` |

## Prerequisites

- Python 3.10+ with CUDA-capable Torch for GPU runs.
- Runtime packages from `skill_manifest.yaml`, especially `monai==1.4.0`, `numpy<2`, `nibabel`, `scipy`, `typer`, `PyYAML`, `fire`, `pytorch-ignite`, `einops`, and `huggingface_hub`. Install `mlflow>=2.10,<4` when MLflow tracking is enabled.
- Optional environment variables: `CUDA_VISIBLE_DEVICES` restricts visible GPUs; `NPROC_PER_NODE` overrides GPU count and values `>=2` select multi-GPU mode for non-sanity runs; `NVSEG_FINETUNE_AUTO_VENV=0` disables the cached MONAI 1.4 compatibility environment. Remote tracking may use `DATABRICKS_CONFIG_PROFILE`, `DATABRICKS_HOST`, `DATABRICKS_TOKEN`, `MLFLOW_TRACKING_CLIENT_CERT_PATH`, `MLFLOW_TRACKING_INSECURE_TLS`, `MLFLOW_TRACKING_PASSWORD`, `MLFLOW_TRACKING_SERVER_CERT_PATH`, `MLFLOW_TRACKING_TOKEN`, or `MLFLOW_TRACKING_USERNAME`; these variables are forwarded only when MLflow is explicitly enabled, and unrelated credentials are not forwarded.
- `--softmax` also needs the pinned NVIDIA-Medtech source checkout. Set `NV_SEGMENT_CT_ROOT` to its `NV-Segment-CT` directory, or set `NV_SEGMENT_CTMR_ROOT` to the sibling `NV-Segment-CTMR` directory. The wrapper reads the official softmax config and implementation in place and writes generated overrides only under `--output-dir`.
- Run outputs: generated bundle configs under `skills/nv-segment-ct-finetune/bundle/configs/`, including `auto_override.json`, `train_continual_task06_lung.json`, and `dfw_no_logging.json`; checkpoints/evidence under `--output-dir`; and local tracking data under `<output-dir>/mlruns` when enabled.
- Dependency cache locations: `~/.cache/nvidia-skills/venvs/nv-segment-ct-finetune-monai14/` for MONAI compatibility packages and `~/.cache/huggingface/` for model assets. These are reusable runtime files, not agent instructions or authorization for another run. Set `NVSEG_FINETUNE_AUTO_VENV=0` when compatibility-environment setup is not approved; then use a caller-provided compatible environment.
- Network access: model/config downloads use `https://huggingface.co` and `https://raw.githubusercontent.com`; remote tracking contacts only the caller-approved MLflow or Databricks destination when explicitly enabled. The label-dictionary download accepts HTTPS on the pinned source host and rejects redirects.

Fresh environment setup:

```bash
python -m pip install "monai==1.4.0" "numpy<2" pytorch-ignite einops nibabel scipy typer PyYAML fire huggingface_hub
```

When MLflow tracking is enabled, also install:

```bash
python -m pip install "mlflow>=2.10,<4"
```

Known upstream compatibility constraints:

- DFW Task06 reference: Python `3.10.16`, MONAI `1.4.0`, Torch `2.7.0+cu126`.
- Use exact `monai==1.4.0` for smoke, sanity, and evidence runs; MONAI 1.5.x can crash the upstream finetune loss on boolean labels.
- Do not float the dependency as `monai>=1.4,<1.6` in generated commands.
- The softmax workflow keeps the upstream defaults of 100 epochs and learning rate `1e-4` unless the caller overrides them.

One-time source setup for `--softmax`:

```bash
export NV_SEGMENT_CTMR_COMMIT=cb921f5c58837c0f42a713855d68b32af88e1cdd
export NV_SEGMENT_CTMR_CHECKOUT="$HOME/.cache/nvidia-skills/upstreams/NV-Segment-CTMR-cb921f5"
if [ ! -d "$NV_SEGMENT_CTMR_CHECKOUT/.git" ]; then
  git clone https://github.com/NVIDIA-Medtech/NV-Segment-CTMR.git "$NV_SEGMENT_CTMR_CHECKOUT"
fi
git -C "$NV_SEGMENT_CTMR_CHECKOUT" checkout --detach "$NV_SEGMENT_CTMR_COMMIT"
export NV_SEGMENT_CT_ROOT="$NV_SEGMENT_CTMR_CHECKOUT/NV-Segment-CT"
```

## Usage

Smoke-scale workflow check:

```bash
python -m pip install "monai==1.4.0" "numpy<2" pytorch-ignite einops nibabel scipy typer PyYAML fire huggingface_hub && \
python skills/nv-segment-ct-finetune/scripts/run_finetune.py \
  PATH_TO_DATASET \
  --smoke \
  --patch-size '[64,64,64]' \
  --output-dir runs/nvseg_smoke
```

Use the staged dataset as `PATH_TO_DATASET`. For the micro fixture, use `skills/nv-segment-ct-finetune/fixtures/spleen_micro`. Smoke mode proves wiring, config generation, checkpoint loading, and runtime compatibility; it is not a quality bar.

MSD Task06 Lung Tumor sanity reproduction:

```bash
python skills/nv-segment-ct-finetune/scripts/run_finetune.py \
  /path/to/Task06 \
  --sanity \
  --output-dir runs/nvseg_task06_sanity
```

The sanity preset follows the single-GPU DFW recipe: fold-0 validation, label mapping `[[1, 23]]` for `lung tumor`, automatic class-prompt segmentation, patch `[128,128,128]`, 5 epochs, and original-spacing `configs/evaluate.json` scoring before and after training. Expected reference range is pretrained Dice about `0.6697`, training-best Dice about `0.6905`, and fine-tuned formal Dice about `0.6836`.

User-data finetune:

```bash
python skills/nv-segment-ct-finetune/scripts/run_finetune.py \
  --dataset-dir /path/to/dataset \
  --datalist /path/to/datalist.json \
  --target-anatomy "lung tumor" \
  --auto-seg \
  --epochs 5 \
  --patch-size '[128,128,128]' \
  --output-dir runs/nvseg_user_finetune
```

From NVIDIA/skills, Apache-2.0.

More from NVIDIA/skills