deepstream-run-mv3dt

Files (11)

• BENCHMARK.md
• SKILL.md
• evals.json · evals/
• amc-calibration-handoff.md · references/
• custom-dataset.md · references/
• generate-configs.md · references/
• sample-run.md · references/
• setup.md · references/
• visualization-metadata.md · references/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# Skill: Run DeepStream MV3DT

## When to Use This Skill

Activate this skill when the user wants to set up, run, verify, or debug the DeepStream Multi-View 3D Tracking reference app. Typical prompts:

- "set up MV3DT DeepStream"
- "run the 4-camera MV3DT sample"
- "run the 12-camera MV3DT sample"
- "run MV3DT on my synchronized MP4s"
- "I have videos but no calibration; calibrate and run MV3DT"
- "show the BEV visualizer or Kafka metadata"
- "stop MV3DT" / "clean up MV3DT" / "tear down MV3DT"

Do not use this skill for single-view 3D tracking, generic DeepStream app development, or live-stream onboarding unless the user explicitly maps that work to this repo's MV3DT pipeline.

## Examples

- "Deploy the DeepStream MV3DT 4-camera sample and show the OSD and BEV windows."
- "Run the MV3DT 12-camera sample headlessly with RTDETR and save videos."
- "Run MV3DT on synchronized MP4s under /data/mv3dt-demo using PeopleNetTransformer."
- "My custom MV3DT videos do not have calibration; use AutoMagicCalib, then run MV3DT."

## Overview

Operate the Multi-View 3D Tracking reference app in `DeepStream/src/apps/reference_apps/deepstream-tracker-3d-multi-view` using the DeepStream Container path. The skill supports setup, shipped sample runs, custom synchronized MP4 datasets, calibration handoff to AutoMagicCalib, display/headless execution, OSD/BEV outputs, and Kafka metadata inspection.

## Prerequisites

- MV3DT reference app directory on disk under `DeepStream/src/apps/reference_apps/deepstream-tracker-3d-multi-view`; if it is absent, ask before cloning the public DeepStream repo
- Docker with NVIDIA GPU support
- DeepStream container image access
- MV3DT sample datasets, models, custom parser libraries, Kafka, Mosquitto, and `mv3dt_venv` prepared by the repo setup script
- Working X11/VNC display for live OSD/BEV windows, or the saved-output headless path for tiled DeepStream MP4 plus Kafka-derived BEV MP4

## Instructions

### Step 0: Resolve MV3DT App Checkout

The skill can be installed outside the DeepStream repo. Resolve `REPO_ROOT` to the MV3DT app directory, not necessarily the Git top-level directory.

```bash
MV3DT_APP_SUBDIR="src/apps/reference_apps/deepstream-tracker-3d-multi-view"

is_mv3dt_app_dir() {
  test -f "$1/README.md" || return 1
  test -d "$1/config_templates" || return 1
  test -d "$1/scripts" || return 1
  grep -q "Multi-View 3D Tracking" "$1/README.md"
}

GIT_TOP="$(git rev-parse --show-toplevel 2>/dev/null || true)"
CANDIDATES=()
if [ -n "${MV3DT_REPO_ROOT:-}" ]; then CANDIDATES+=("${MV3DT_REPO_ROOT}"); fi
if [ -n "${DEEPSTREAM_REPO_ROOT:-}" ]; then CANDIDATES+=("${DEEPSTREAM_REPO_ROOT}/${MV3DT_APP_SUBDIR}"); fi
CANDIDATES+=("${PWD}")
if [ -n "${GIT_TOP}" ]; then
  CANDIDATES+=("${GIT_TOP}" "${GIT_TOP}/${MV3DT_APP_SUBDIR}")
fi
CANDIDATES+=("${HOME}/DeepStream/${MV3DT_APP_SUBDIR}" "${HOME}/deepstream/${MV3DT_APP_SUBDIR}")

REPO_ROOT=""
for candidate in "${CANDIDATES[@]}"; do
  if [ -n "$candidate" ] && is_mv3dt_app_dir "$candidate"; then
    REPO_ROOT="$(cd "$candidate" && pwd)"
    break
  fi
done

if [ -z "${REPO_ROOT}" ]; then
  cat <<'EOF'
ERROR: MV3DT reference app directory was not found.
Set MV3DT_REPO_ROOT to an existing deepstream-tracker-3d-multi-view app directory, or ask the user to approve cloning the public DeepStream repo and then run:

  DEEPSTREAM_REPO_ROOT="${DEEPSTREAM_REPO_ROOT:-$HOME/DeepStream}"
  git clone https://github.com/NVIDIA/DeepStream.git "$DEEPSTREAM_REPO_ROOT"
  export MV3DT_REPO_ROOT="$DEEPSTREAM_REPO_ROOT/src/apps/reference_apps/deepstream-tracker-3d-multi-view"

Do not clone silently.
EOF
  exit 1
fi

cd "${REPO_ROOT}"
export REPO_ROOT MV3DT_REPO_ROOT="${REPO_ROOT}"
```

If the app directory cannot be resolved, ask the user for an existing checkout path or for approval to clone `https://github.com/NVIDIA/DeepStream`. Do not clone silently.

### Step 1: Select The Primary Workflow

Load exactly one primary reference for the user's current request:

| User intent | Reference |
|---|---|
| Install, prepare, or verify prerequisites | `references/setup.md` |
| Run bundled 4-camera or 12-camera sample | `references/sample-run.md` |
| Run custom synchronized MP4s | `references/custom-dataset.md` |
| Missing custom calibration | `references/amc-calibration-handoff.md`, then return to `references/custom-dataset.md` |
| View OSD, BEV, screenshots, recordings, or Kafka metadata | `references/visualization-metadata.md` |
| Stop a run, clean generated artifacts, or stop prerequisite services | `references/setup.md` |

If setup, datasets, models, Kafka, Mosquitto, Docker GPU runtime, or the Python venv are missing, load `references/setup.md` before continuing to the user's original workflow.

When `sample-run.md` or `custom-dataset.md` needs to regenerate DeepStream configs, load `references/generate-configs.md` as the canonical shared config-generation reference rather than duplicating the shell logic.

### Step 2: Follow The Run Stages

For every run, use this stage order:

| Stage | Action |
|---|---|
| Validate | Check app directory, prerequisites, dataset shape, display/headless mode, Docker GPU support, and output-directory writability. |
| Prepare | State the selected sample/custom dataset, detector, run mode, output directory, expected output surfaces, and the Docker security decision for `--privileged --net=host` before launch. |
| Execute | Generate configs, start offline BEV capture first in headless mode or when saved BEV MP4 is explicitly requested, then run DeepStream or delegate missing calibration to AutoMagicCalib. |
| Verify | Confirm functional readiness with `App run successful`, fresh MP4 artifacts when file output is enabled, Kafka offsets/messages, and BEV message/frame counts when BEV MP4 capture runs. |
| Report | Summarize selected options, generated files, artifact paths, file sizes/counts, and any skipped or failed output surface; after a default 4-camera sample run, mention that the 12-camera sample is also available as a follow-up. |

### Step 3: Apply Defaults Explicitly

From NVIDIA/skills, CC-BY-4.0 AND Apache-2.0.

More from NVIDIA/skills