foundationpose-pipeline

Files (9)

• BENCHMARK.md
• SKILL.md
• openai.yaml · agents/
• config.yml · evals/
• evals.json · evals/
• baseline.json · evals/files/
• candidate.json · evals/files/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# Run and evaluate the FoundationPose perception pipeline

## Purpose

Run depth, SAM3 segmentation, and FoundationPose on BOP-format datasets using the TAO Deploy
TensorRT depth engine. Adapt datasets, preserve run provenance, and interpret pose metrics.
For missing dependencies or engine construction, use `foundationpose-setup` if installed,
or the product checkout's README Install and Verify sections.

## Requirements

Locate the user's [product checkout](https://github.com/nvidia-isaac/foundationpose_perception_pipeline)
by `pyproject.toml` (project `foundationpose-perception-pipeline`), `script/run_pipeline.py`, and
`config/defaults.yaml`. Run commands from that root, not from this installed skill's directory.
A catalog install supplies instructions, not the product code, datasets, or weights. If execution
was requested and no checkout exists, obtain it from the URL above and complete setup first.
For advice or analysis of supplied artifacts, use those inputs without cloning or loading models.

Execution requires the product's Python 3.12 venv, authorized SAM3 checkpoint access, the built
FoundationPose library, an adapted dataset, a matching TAO engine with its sidecar, and sufficient
GPU memory. Read the checkout's README Configuration and Dataset adaptation sections for
profile paths; read ARCHITECTURE.md Outputs for the matching artifact schema.

Set absolute paths before GPU work:

```bash
export FOUNDATIONPOSE_ROOT="$(realpath ../foundation-pose-inference-library)"
PIPELINE_SITE="$(realpath .venv/lib/python3.12/site-packages)"
export LD_LIBRARY_PATH="${PIPELINE_SITE}/tensorrt_libs:${PIPELINE_SITE}/nvidia/cu13/lib:${LD_LIBRARY_PATH:-}"
./.venv/bin/python -c "import ctypes; ctypes.CDLL('libcudart.so.13'); print('ok')"
```

Do not mix libraries from another venv into this path. Skipping the check can cause pose to fail
after depth has already completed.

## Instructions

### 1. Resolve the task and inputs

Identify the profile, dataset name, source or adapted scene paths, engine, ground-truth
availability, and output directory. `<profile>` and `<dataset>` may differ. `--config` selects a
profile; it does not replace a required `--dataset`. Same-named profiles can be inferred by
commands that take `--dataset`.

| Request | Entry point |
|---|---|
| Convert a supported BOP dataset | `tools/bop_adapt/adapt.py` |
| Inference without pose ground truth | `script/infer.py` |
| Inference plus scoring | `script/run_pipeline.py` |
| Score a completed run with new scoring parameters | `script/evaluate.py` |
| Sweep several datasets | `script/run_batch_eval.py` |

A capture without `scene_gt.json` can use inference only. `--no-depth-metrics` skips collected
sensor-depth scoring; it does not remove the pose-ground-truth requirement for evaluation.

### 2. Adapt before building or checking an engine

Skip adaptation only for the pipeline's rig layout:
`<split>/<scene>/rgb/<im_id>.png`, one `scene_camera.json` per scene, and im_ids representing
rig cameras (base camera 0 in the shipped profiles).

```bash
./.venv/bin/python tools/bop_adapt/adapt.py --config <profile> --src <downloaded-dataset>
```

The profile's `dataset.name` selects a registered adapter; `--help` exposes its flags. An unknown
adapter is not supported automatically. On static-scene datasets the adapter emits one scene
per usable (source scene, base frame) pair and reports skipped frames with no rectifiable partner.
Changing the baseline band changes the adapted data: rebuild GT caches and regenerate depth.

Engine building uses `tools/build_tao_engine.py --shape-from-scene <adapted-scene>`; see the
checkout's README Install section if the setup skill is unavailable. A raw BOP directory or raw
image dimensions do not establish the required rectified engine shape.

### 3. Validate inputs and prepare GT caches

```bash
./.venv/bin/python test/check_engine_depth_smoke.py \
  --config <profile> --dataset <dataset> --engine <engine-path>
```

Expect `backend=tao`, `normalization=imagenet`, a fixed shape, a plausible valid fraction, and no
`cropping N rows` warning. A stale sidecar, changed GPU/TensorRT/precision, changed max-width,
or cropping requires rebuilding the engine and regenerating depth. Do not bypass these checks.

For scoring runs with pose GT, precompute the cache:

```bash
./.venv/bin/python script/build_gt_cache.py --config <profile> --dataset <dataset>
```

Use `--config <profile> --all` for all matching datasets. Missing collected depth calls for
`--no-depth-metrics`; missing `scene_gt.json` calls for inference only. Check the resolved
`dataset.collected_depth_root` using the actual path, not a shell command substitution.

### 4. Run only the work needed

For a new end-to-end run:

```bash
./.venv/bin/python script/run_pipeline.py --config <profile> --dataset <dataset> \
  --output-dir output/<new-run> --foundation-stereo-model <engine-path> \
  --depth-backend commercial --no-depth-metrics
```

Omit `--no-depth-metrics` when collected sensor depth is available and should be scored.
Start with `--max-scenes 1` for a time/fit check before sizing a larger run.
For a capture with no pose ground truth:

```bash
./.venv/bin/python script/infer.py --config <profile> --dataset <dataset> \
  --output-dir output/<new-run> --foundation-stereo-model <engine-path> \
  --depth-backend commercial
```

The model path selects the backend. `--depth-backend commercial` asserts that selection; it
neither downloads a model nor establishes rights to the weights. Set the engine once in the
profile's `overrides.depth.engine` to avoid repeating the model-path flag.

From NVIDIA/skills, Apache-2.0.

More from NVIDIA/skills