detection-training

Files (1)

• SKILL.md

SKILL.md (first lines)

# Detection training (LanceDB view → Faster R-CNN)

This tool trains an object detector on a **slice** of a dataset rather than the
whole thing. The input is a LanceDB materialized view, which is what makes the
failure-mode workflow possible: curate a view of nighttime frames, or distant
objects, or riders, then train and evaluate only on that slice.

Upstream is `skills/tools/lancedb/SKILL.md` — create the view with
`npa workbench lancedb create-mv` / `refresh-mv` before anything here. The end-to-end
pipeline is `workflows/testing/bdd100k-pipeline.yaml`, with a
walkthrough in `docs/workbench/cookbooks/bdd100k-pipeline.md`.

## Two execution modes

Direct (the default) runs the work from your CLI invocation. Service mode calls a
deployed Kubernetes endpoint:

```bash
npa workbench detection-training train --view <mv-name> --output-uri s3://<bucket>/runs/<id>/
npa workbench detection-training train --view <mv-name> --service --endpoint <url>
```

Deploy the service only when you want a persistent endpoint several runs share:

```bash
npa workbench detection-training deploy \
  --project <alias> --cluster-name <name> \
  --input-path s3://<bucket>/lancedb/<db>/ \
  --output-path s3://<bucket>/detection/ \
  --gpu-type h100 --namespace default \
  --dry-run                       # prints the manifest without applying
npa workbench detection-training deploy --project <alias> --destroy
```

`--gpu-type` accepts `h100`, `b200`, `l40s`, and `rtxpro6000`; verify actual cluster labels and reservation placement before deploying. Auth defaults to `token` (the token comes from
the variable named by `--token-env`, default `DETECTION_TRAINING_TOKEN`);
`--insecure-no-auth` exists but should not be used. Official NPA GHCR images
pull anonymously, so `--image-pull-secret` defaults to empty. For an optional
operator-controlled private registry, pass the name of an existing standard
Kubernetes pull secret explicitly. Always `--dry-run` first and read the
manifest.

## Train

```bash
npa workbench detection-training train \
  --view <materialized-view> \
  --output-uri s3://<bucket>/detection/runs/<id>/ \
  --label-map '{"person":0,"rider":1,"car":2}' \
  --epochs 10 --batch-size 8 --learning-rate 0.005 \
  --wait --poll-seconds 30 --timeout-seconds 21600 \
  --output json
```

`--view` is the only required flag. `--data-path` (alias `--lance-uri` /
`--input-path`) overrides which LanceDB the view is read from. `--override
KEY=VALUE` sets request fields the flags do not expose. Weights and Biases
logging is opt-in and defaults to `offline` mode.

`--wait` polls `/status` until the run completes **and fails if it does not** —
without it, the command returns as soon as the run is accepted, which is easy to
misread as success. Default output format for `train` and `eval` is `json`.

## The label map is not optional for BDD100K

**`--label-map` is required whenever the dataset stores string categories, which
BDD100K does.** Without it, category names reach `int()` and the run dies on:

```
invalid literal for int() with base 10: 'train'
```

`train` is a real BDD100K category — the vehicle. Reading that traceback as a
bug in the training code, rather than a missing label map, costs more time than
it should. Accepted formats are JSON (`{"person":0,...}`) or comma-separated
pairs (`person=0,rider=1`). Class count is inferred from the map by default.
Source maps containing zero shift all IDs by one inside the detector because
Faster R-CNN reserves zero for background. Checkpoints retain both maps; string
and numeric annotations from materialized views keep the same category identity.
Explicit `--num-classes` includes background and must cover every mapped ID.

## Evaluate

```bash
npa workbench detection-training eval \
  --checkpoint-uri s3://<bucket>/detection/runs/<id>/ \
  --eval-view <validation-mv> \
  --output-uri s3://<bucket>/detection/eval/<id>/ \
  --label-map '{"person":0,"rider":1,"car":2}' \
  --discover-checkpoint \
  --write-canonical-metrics
```

`--eval-view` and `--output-uri` are required. `--discover-checkpoint` changes
what `--checkpoint-uri` means: it becomes the training **output prefix** to
search, and the checkpoint is selected from the last completed `/runs` entry
using its verified final checkpoint artifact. Legacy records without artifact
metadata retain the epoch-pattern fallback. Use it instead of hand-assembling a
checkpoint filename from an epoch count you assumed.

`--write-canonical-metrics` publishes the response under the output prefix so
downstream stages and `npa workbench insights` can pick it up. Evaluation reads
the checkpoint map automatically. An explicit map must match its category
identity; mismatches fail before model execution. Legacy checkpoints preserve
the original IDs. Missing metric dependencies fail instead of returning fake
zero scores; an undefined COCO metric remains `-1`.

## In workflows

toolRefs are per-slice pairs rather than one generic tool:
`workbench.detection_training.train_nighttime` / `.eval_nighttime`,
`.train_distant` / `.eval_distant`, `.train_rider` / `.eval_rider`. They appear
together in `bdd100k-pipeline.yaml`, which builds the views, trains each slice,
and evaluates each against its validation view.

## Gotchas

- **Train and eval must agree on category identity.** New checkpoints record it
  and evaluation rejects a conflicting explicit map.

From nebius/nebius-physical-ai, Apache-2.0.

More from nebius/nebius-physical-ai