jetson-llm-benchmark

Files (8)

• BENCHMARK.md
• SKILL.md
• evals.json · evals/
• bench_llama_cpp.sh · scripts/
• bench_ollama.sh · scripts/
• bench_vllm.sh · scripts/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# Jetson LLM Benchmark

Reproducible Jetson benchmarks with **structured JSON output** so an agent can compare runs. Encodes the workflow from the [Jetson AI Lab GenAI Benchmarking tutorial](https://www.jetson-ai-lab.com/tutorials/genai-benchmarking/).

## Purpose

Measure deployed LLM latency and throughput on a Jetson target using the correct
runtime-specific benchmark wrapper. Use the JSON output to compare models,
runtime flags, power modes, and before/after tuning changes.

## Prerequisites

- Run on the Jetson device that hosts the model runtime.
- For vLLM, start the OpenAI-compatible vLLM server first and know the served
  model ID.
- For Ollama, ensure the Ollama daemon is reachable at `--endpoint` and the
  named model is already pulled.
- For llama.cpp/GGUF, provide a readable `.gguf` model path on the host.
- Put the device in the intended power mode before measuring. MAXN is preferred
  for comparable performance numbers.

## Available Scripts

| Script | Purpose | Arguments |
|--------|---------|-----------|
| `scripts/bench_vllm.sh` | Runs `vllm bench serve` against a running OpenAI-compatible vLLM server. | `--model`, `--endpoint`, `--concurrency`, `--input-len`, `--output-len`, `--num-prompts`, `--no-warmup`, `--container`, `--native`. |
| `scripts/bench_llama_cpp.sh` | Runs `llama-bench` for a local GGUF model through the Jetson-appropriate NVIDIA-AI-IOT llama.cpp container. | `--model`, `--n-prompt`, `--n-gen`, `--n-gpu-layers`, `--threads`, `--container`. |
| `scripts/bench_ollama.sh` | Benchmarks a local or containerized Ollama daemon through the `/api/generate` REST API. | `--model`, `--endpoint`, `--num-prompts`, `--input-len`, `--output-len`, `--no-warmup`. |

If your agent runtime supports `run_script`, invoke the selected wrapper directly with the user-provided model identifier or local model path, then summarize the returned JSON. Otherwise run the wrapper with `bash {baseDir}/scripts/<wrapper-name> ...`.

## Instructions

Always use the matching wrapper script for the runtime — do **not** call the underlying `vllm bench serve`, `llama-bench`, or `curl` against `/api/generate` by hand:

- vLLM → `scripts/bench_vllm.sh` (required for the vLLM path)
- llama.cpp / GGUF → `scripts/bench_llama_cpp.sh` (required for the GGUF path)
- Ollama → `scripts/bench_ollama.sh` (required for the Ollama path)

These wrappers handle warmup, the NVIDIA-AI-IOT container selection, and JSON emission. Calling the underlying tool directly will not satisfy the output contract below.

For "how do I benchmark/measure" questions, first run the matching wrapper with
`--help` to verify the exact options, then answer with the wrapper command. Do
not run a full benchmark unless the user asks you to execute it or the required
server/model path is already confirmed.

## Expected Workflow

Pick exactly one wrapper based on the runtime the user named, and invoke that
wrapper with `--help` before composing the answer. Do not merely mention the
script name. If the runtime does not execute scripts relative to the skill
directory, use `{baseDir}/scripts/<wrapper-name>`.

- Existing vLLM OpenAI-compatible server at `localhost:8000`:
  `{baseDir}/scripts/bench_vllm.sh --help`, then show a command using
  `--concurrency 1,8` and the served model ID.
- llama.cpp / GGUF / `llama-server`: `{baseDir}/scripts/bench_llama_cpp.sh
  --help`, then show a command for the GGUF model path and report that
  prompt/generation speed maps to TTFT, ITL/TPOT, and throughput.
- Ollama: `{baseDir}/scripts/bench_ollama.sh --help`, then show a command with
  `--model <ollama-tag>`. Do not use vLLM or llama.cpp wrappers for Ollama.

## When to use

- "Benchmark / measure / compare X on this Jetson."
- After `jetson-llm-serve` to actually quantify the deployment.
- Before/after applying flags from `jetson-inference-mem-tune` to confirm the change helped.

## Three paths — pick by runtime

### A. vLLM (preferred for parity with how things are served)

Server must already be running (use `jetson-llm-serve`). Run **`bench_vllm.sh`**:

```bash
scripts/bench_vllm.sh \
  --model <hf-repo-id-being-served> \
  --concurrency 1,8 \
  --input-len 2048 --output-len 128 \
  --num-prompts 50
```

Uses the Jetson-appropriate benchmark client path: upstream vLLM 0.20+ container
`vllm/vllm-openai:latest` on Thor and Orin JetPack 7.2 / L4T r39+,
or the NVIDIA-AI-IOT vLLM benchmark container
`ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin` on older Orin. Pass
`--native` only when host-native vLLM is already installed and validated. It
runs against `http://localhost:8000/v1`. **Always do a warmup pass first** (~10
prompts, discarded) before the measured run — Jetson has cold caches and JIT'd
kernels.

### B. Ollama (for models served by a running Ollama daemon)

No benchmark container needed. Uses Ollama's `/api/generate` REST API directly —
timing data (TTFT, ITL, throughput) comes from the response JSON, so no
`--verbose` parsing is required.

**Prerequisite:** the Ollama daemon must be reachable at `--endpoint` (default
`http://localhost:11434`). This works whether Ollama is installed natively or
running in a container that exposes that port. If the daemon is not running,
the script will tell you whether Ollama is installed but stopped (`ollama serve`
to fix) or not installed at all (install instructions printed). Run
**`bench_ollama.sh`** (do not roll your own `curl` against `/api/generate`):

```bash
scripts/bench_ollama.sh \
  --model <ollama-model-name> \
  --num-prompts 20 \
  --input-len 512 --output-len 128
```

Runs sequential single-stream requests (concurrency=1). Ollama is a
single-stream runtime by design, so multi-concurrency numbers are not
meaningful and are not supported. Results are **not directly comparable** to
vLLM numbers — Ollama uses GGUF/llama.cpp internals while vLLM uses its own
CUDA kernels.

### C. llama.cpp (for GGUF models)

From NVIDIA/skills, Apache-2.0.

More from NVIDIA/skills