deepstream-sop

Files (58)

• BENCHMARK.md
• SKILL.md
• actions.json · configs/
• nvds_inference_template.txt · configs/
• nvds_preprocess_template.txt · configs/
• vlm_prompts.txt · configs/
• evals.json · evals/
• Dockerfile_reference · references/
• Emulation_0815-0000.pfs · references/
• Makefile_custom_postprocess_reference · references/
• README_reference.md · references/
• compose_reference.yaml · references/
• copy_sources_reference.sh · references/
• ddm_net_reference.py · references/
• ddm_pytorch2.patch · references/
• eval_sop_prompt.md · references/
• example_sop_prompt.md · references/
• export_ddm_to_tensorrt_reference.py · references/
• ext.proto · references/
• ext_pb2_reference.py · references/
• license.txt · references/
• messager_reference.py · references/
• missing_number_detector_reference.py · references/
• nv.proto · references/
• nv_pb2_reference.py · references/
• nvds_custom_postprocess_tensor_reference.cpp · references/
• package_file_list_reference.txt · references/
• protos_init_reference.py · references/
• skill_01_fastapi_endpoints.md · references/
• skill_02_pydantic_schemas.md · references/
• skill_03_deepstream_pipeline.md · references/
• skill_04_config_templates.md · references/
• skill_05_triton_ddm_model.md · references/
• skill_05b_custom_postprocess.md · references/
• skill_06_sop_process_manager.md · references/
• skill_06b_sop_checker.md · references/
• skill_07_sse_streaming.md · references/
• skill_08_basler_camera.md · references/
• skill_09_docker_build_deploy.md · references/
• skill_10_test_suite.md · references/
• skill_11_env_variables.md · references/
• skill_12_evaluation_workflow.md · references/
• skill_13_verification_curl.md · references/
• skill_14_implementation_checklist.md · references/
• skill_15_latency_measurement.md · references/
• skill_16_message_schema.md · references/
• skill_17_camera_latency_measurement.md · references/
• skill_18_rtsp_streaming_output.md · references/
• sop_architecture.svg · references/
• sop_step_checker_reference.py · references/
• start_server.sh · references/
• test_api_endpoints_reference.py · references/
• triton_config_template.pbtxt · references/
• triton_model_reference.py · references/
• utils_reference.py · references/
• vllm_inference_reference.py · references/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# DeepStream SOP Inference Microservice Skill

This skill guides AI coding assistants in building, extending, and debugging the
**NVIDIA DeepStream SOP (Standard Operating Procedure) Inference Microservice** —
a GPU-accelerated pipeline for temporal action detection and VLM-based SOP compliance
monitoring on industrial video feeds.

**Reference repository**: https://github.com/NVIDIA/sop-monitoring-blueprints/tree/main/microservices/sop-inference-bp
**Local reference code**: `sop-inference-bp/` directory (from a local clone of the repository)

---

## Models

Model-agnostic at both inference stages — swap via env var (and Triton dir for GEBD).

| Stage | Role | Model class | Default | Swap via |
|------|------|-------------|---------|----------|
| Stage 1 (CV) | Per-frame boundary scoring → chunk segmentation | **Generic Event Boundary Detection (GEBD)** | **DDM** ([MCG-NJU/DDM](https://github.com/MCG-NJU/DDM)) via Triton Python backend | Replace `triton_model_repo/<model>/` + `DDM_MODEL_PATH` (§ 5) |
| Stage 3 (VLM) | Per-chunk action classification | Vision-language model via vLLM | **Cosmos Reason 1 7B** (Reason 2 also supported) | Set `VLLM_MODEL_PATH` to a different HF ID or local path |

"GEBD" = swappable Stage-1 slot; "DDM" = the default architecture (terms used interchangeably).

**Chunking is selectable per request** (§ 2): default `ddm-net` uses GEBD; `uniform` produces fixed-length chunks and **bypasses Stage-1 GEBD** (§ 3, § 6). DDM temporal window is configurable via `FRAMES_PER_SIDE` / `SEQUENCE_BATCH` (§ 4, § 5), with optional **TensorRT** (§ 5).

---

## Architecture Overview

Runs in a Docker container (`nvds-action-sop`) alongside a Kafka container. Full diagram: [`references/sop_architecture.svg`](references/sop_architecture.svg).

**Data flow through the 4-stage `SOPVideoProcessor` pipeline (per-request):**

```
Input Sources                    Docker Container: nvds-action-sop
─────────────                    ──────────────────────────────────────────────────
Video Files ──┐                  FastAPI Server (port 8300)
RTSP Streams ─┤── base64/       ├─ /v1/chat/completions → SOPProcessManager
Basler Camera ┘   file/rtsp/       │
                  camera           │ ModelInitializer: VLM first, then DDM dummy pipeline
                                   │ 4 Thread Pools: cv(32), clip(32), vlm(64), vlm_req(64)
                                   │
                                   ▼ SOPVideoProcessor (per-request)
                                   ┌────────────────────────────────────────────────┐
                                   │ Stage 1: DeepStream Pipeline (GPU)             │
                                   │   Source → nvstreammux → tee1                  │
                                   │    ├─[inference] queue1 → nvdspreprocess       │
                                   │    │  → nvinferserver (Triton CAPI + DDM)      │
                                   │    │  → InferOutputTensorParser → score_queue  │
                                   │    ├─[frames]  queue3 → nvvideoconvert         │
                                   │    │  → capsfilter → appsink                   │
                                   │    │  → DecodedFrameRetriever → frame_queue    │
                                   │    └─[RTSP out] queue → convert → H.264 enc    │  (optional, § 18)
                                   │       → rtppay → udpsink → RTSPServer (§ 18)   │  opt-in only
                                   │              │ boundary scores                 │
                                   │              ▼                                 │
                                   │ Stage 2: Clip Post-Process                     │
                                   │   Boundary detection → chunk segmentation      │
                                   │              │ video frames + timestamps        │
                                   │              ▼                                 │
                                   │ Stage 3: VLM Inference                         │
                                   │   Embedded vLLM (Cosmos Reason 1/2)            │
                                   │   Frame sampling at VLM_FPS → classification   │
                                   │              │ action labels                    │
                                   │              ▼                                 │
                                   │ Stage 4: SOP Checker                           │
                                   │   Sequence validation → missing/misordered     │
                                   │              │ chunk results                    │
                                   │              ▼                                 │
                                   │         final_queue                            │
                                   └────────────────────────────────────────────────┘
                                          │
Output                                    ▼
──────                             ┌─────────────────┐
SSE Stream (chat.completion.chunk) │ Kafka Messages   │
Non-streaming (chat.completion)    │ (JSON/Protobuf)  │
Prometheus metrics (/v1/metrics)   └────────┬────────┘
                                            ▼
                                   Docker Container: kafka
                                   (apache/kafka:3.7.0)
```

---

## Section Index

Each section is a standalone file in `references/` — load only what your task needs.

| § | File | Responsibility |
|---|------|---------------|
| 1 | [`skill_01_fastapi_endpoints.md`](references/skill_01_fastapi_endpoints.md) | FastAPI endpoints, server init, Prometheus metrics |
| 2 | [`skill_02_pydantic_schemas.md`](references/skill_02_pydantic_schemas.md) | Request/response Pydantic models (`api_types.py`) |
| 3 | [`skill_03_deepstream_pipeline.md`](references/skill_03_deepstream_pipeline.md) | DeepStream pyservicemaker pipeline, tensor parser, dummy pipeline |
| 4 | [`skill_04_config_templates.md`](references/skill_04_config_templates.md) | nvdspreprocess / nvinferserver config templates + rendering |
| 5 | [`skill_05_triton_ddm_model.md`](references/skill_05_triton_ddm_model.md) | Triton model repo, config.pbtxt, model.py, ddm_net.py |
| 5b | [`skill_05b_custom_postprocess.md`](references/skill_05b_custom_postprocess.md) | C++ postprocess plugin, Makefile, IOptions API |
| 6 | [`skill_06_sop_process_manager.md`](references/skill_06_sop_process_manager.md) | SOPProcessManager, SOPVideoProcessor, VLLMInference, Kafka |
| 6b | [`skill_06b_sop_checker.md`](references/skill_06b_sop_checker.md) | SOP sequence and checker compliance: MissingNumberDetector, SopCheckerCache, SopCheckerRequest/Response |
| 7 | [`skill_07_sse_streaming.md`](references/skill_07_sse_streaming.md) | SSE generator, stream response formatting, dummy test mode |
| 8 | [`skill_08_basler_camera.md`](references/skill_08_basler_camera.md) | Basler camera support, Pylon SDK, emulation, formats |
| 9 | [`skill_09_docker_build_deploy.md`](references/skill_09_docker_build_deploy.md) | Docker build, deploy, .env configuration |
| 10 | [`skill_10_test_suite.md`](references/skill_10_test_suite.md) | Test suite coverage, assertions, running tests |
| 11 | [`skill_11_env_variables.md`](references/skill_11_env_variables.md) | All environment variables reference |
| 12 | [`skill_12_evaluation_workflow.md`](references/skill_12_evaluation_workflow.md) | End-to-end eval workflow: static checks, build, launch, tests, API/camera/Kafka checks, report |
| 13 | [`skill_13_verification_curl.md`](references/skill_13_verification_curl.md) | Verification steps and curl examples |
| 14 | [`skill_14_implementation_checklist.md`](references/skill_14_implementation_checklist.md) | Implementation checklist: file copy list, generated files, Docker prereqs, verification |
| 15 | [`skill_15_latency_measurement.md`](references/skill_15_latency_measurement.md) | TTFC and C2C latency measurement for file input via SSE streaming |
| 16 | [`skill_16_message_schema.md`](references/skill_16_message_schema.md) | Kafka message schema selection (`JSON` default vs `NvProtoSchema`) and extending messages with custom data |
| 17 | [`skill_17_camera_latency_measurement.md`](references/skill_17_camera_latency_measurement.md) | Camera / live-stream chunk_e2e latency measurement using internal pipeline timestamps |
| 18 | [`skill_18_rtsp_streaming_output.md`](references/skill_18_rtsp_streaming_output.md) | **OPT-IN** RTSP streaming output: `tee1`-tap re-stream, `RTSPStreamingServer`, `SW_ENCODER` toggle. Generate **only** when user explicitly requests RTSP |

For end-to-end evaluation, read § 12 first; load build/test/curl/latency/camera/Kafka as needed.

**§ 18 is opt-in** — generate only when the user explicitly requests RTSP output; otherwise skip § 18 and the `RTSP_*` rules below.

---

## Key Files Map

The full source-to-target file mapping lives in

From NVIDIA/skills, CC-BY-4.0 AND Apache-2.0.

More from NVIDIA/skills