serving-llms-on-epyc

Files (12)

• .federated.json
• SKILL.md
• epyc.json · data/
• evals.json · evals/
• machine.yml · evals/
• reference.md
• check_model.py · scripts/
• cpu_tune.py · scripts/
• detect.py · scripts/
• estimate_memory.py · scripts/
• validate.py · scripts/
• skill-card.md

SKILL.md (first lines)

# Serving LLMs on AMD EPYC™ (vLLM + zentorch, CPU)

Bring up a single vLLM OpenAI endpoint on an AMD EPYC™ host with the zentorch CPU
backend, sized to the hardware. Container-first (Docker or Podman); conda/host
is the fallback. An installed AMD Instinct GPU does not disqualify the host:
select this skill when the endpoint itself should run on the EPYC CPU.

**This is single-socket serving:** one instance pinned to one socket and its memory
(vLLM scales poorly across sockets, so we do not span them). On a dual-socket host it
runs on a single socket; the multi-socket answer is **multiple instances (one per
socket)**, which is out of scope for this single-instance recipe.

Hard rule for this skill: **on any failure, report the cause + logs and STOP.
Do not retry, do not debug.** (Debugging is a separate workflow.)

**The agent does the serve flow itself** -- pull, configure, launch, poll --
using the runtime `validate.py` reports. Never hand the user per-serve commands.
Like serving-llms-on-instinct, an accessible container runtime is a one-time
**prerequisite**: if `validate.py` finds none, report its one-time fix (make
docker accessible / install podman / provide a conda env) and stop. Do not
attempt `sudo` or privilege escalation.

## Data file

Read `data/epyc.json` directly. It holds the container image, mandatory CPU run
flags, supported precision, the model-support policy, the default model, and the
verified throughput-flag gotcha. Its `vllm_version` and image tag are one
validated default stack; keep them aligned and do not hardcode either from memory.

## Step 1: Detect the CPU

```bash
python3 scripts/detect.py            # add --host user@box for a remote host
```

Returns `cpu_model`, `is_amd_epyc`, `epyc_generation`
(Naples/Rome/Milan/Genoa/Bergamo/Siena/Turin/Venice or EPYC 4004/4005),
`zen_arch`, `is_supported_epyc`, `avx512`, `logical_cores`, `physical_cores`,
`sockets`, `numa_nodes`, `memory_gb`.

Route from detect.py -- decide the serving path:
- `is_amd_epyc` is `false` -> stop: this skill targets AMD EPYC. (Other x86 may work
  but is unsupported here.)
- `avx512` is `false` -> **zentorch cannot run** on this CPU (its bf16 path needs
  AVX-512 BF16, `avx512_bf16`, which only lands on Zen4+). This is a pre-Zen4 EPYC
  (Naples / Rome / Milan, 7000 series). Do **not** dead-end -- it is still an EPYC
  host: **offer the stock vLLM CPU path** (plain vLLM, no zentorch acceleration --
  slower, but verified working on EPYC 7763/Milan). Proceed only on the user's explicit
  OK, launching the official stock vLLM CPU image (Step 6 "Stock vLLM" variant); if the
  user declines, stop.
- `is_supported_epyc` is `false` but `avx512` is `true` (e.g. Bergamo / Siena /
  EPYC 4004/4005) -> the zentorch path is **not validated** for this generation. Get
  explicit confirmation to try zentorch unvalidated, or take the same stock vLLM offer.
- else (**9000 series** -- Genoa/Turin/Venice -- with AVX-512 BF16) -> the validated
  zentorch path. Proceed.

`validate.py` (Step 2) reports `zentorch_capable` and sets `requires_confirmation`
for the stock/unvalidated paths, so this routing is enforced there too.

Carry `epyc_generation` / `avx512` through the later phases -- e.g. Venice packs up
to 256 cores/socket, which the thread-binding in Step 5 sizes from.

## Step 2: Validate the runtime and environment

```bash
python3 scripts/validate.py --image <image from data/epyc.json> --generation <epyc_generation from detect> --avx512 <avx512 from detect>
```

This also **hard-enforces the AVX-512 gate**: on a CPU without AVX-512, validate.py
returns a blocking `error` (`ready: false`) so the flow stops here regardless of the
Step 1 prose -- no image pull, no launch. (It reads the local CPU itself if `--avx512`
is omitted, so the gate holds even if the value was not passed through.)

Returns `ready`, `requires_confirmation`, `zentorch_capable` (false -> zentorch
can't run; use the stock vLLM variant in Step 6), `runtime` (`docker`, `podman`, or
null), `runtime_detail`, `conda_path_available`, `stack`, `compatibility`, `hf_cache`
(resolved HF cache mount -- use `hf_cache.mount` at launch), `ram_gb`, and
`errors/warnings/advisories`. If `requires_confirmation` is set (stock/unvalidated
path), surface that and get the user's OK before launching. Pick the path:
- `runtime` is `docker` or `podman` -> container path (Step 6), used verbatim.
- `runtime` null but `conda_path_available: true` -> conda/host path.
- `runtime` null and no conda -> `ready` is false. Report the one-time
  onboarding `fix` (make docker accessible / install podman / conda env) and stop.

Do not proceed if `ready` is `false`.

**Stack-compatibility gate.** `validate.py` probes the *selected* runtime for its
exact `vllm`/`zentorch`/`torch` versions and the active vLLM platform, then sets
`compatibility.status`:
- `proceed` -> the stack is the validated default (or a validated family on a Zen
  platform); continue.
- `blocked` -> a stock CPU platform is active, so zentorch acceleration is **not**
  on (error). Report `compatibility.message` and stop.
- `confirmation_required` (`requires_confirmation: true`) -> **Venice on a vLLM
  other than the pinned default**. This recipe has not been validated on Venice
  with that version. Surface `compatibility.message`, recommend the pinned
  `vllm_version` image from `data/epyc.json`, and **stop for an explicit user
  go/no-go** before launching. On the pinned default vLLM, Venice proceeds with no
  warning.

The gate only runs once the image is local. If `validate.py` reports the image is
not pulled, pull it (or let Step 6 pull it) and **re-run `validate.py`** so the
gate probes the real stack rather than only the tag.

## Step 3: Resolve and validate the model

If the user named no model, use `default_model` from `data/epyc.json`
(`Qwen/Qwen3-0.6B` -- ungated, tiny, fast first success). Otherwise use theirs.

Check that vLLM actually supports the model (do **not** blanket-block multimodal).
Pass the vLLM version the model will actually run on: use `stack.vllm` from
`validate.py` when it was probed (the conda env may differ from the pin), else the
`vllm_version` from `data/epyc.json`.

```bash
python3 scripts/check_model.py --model-id <model> --revision <rev or main> --vllm-version <stack.vllm from validate, else vllm_version from data/epyc.json>
```

- Exit 0 = vLLM serves it as a generation endpoint, or support is undeterminable

From amd/skills, MIT.

More from amd/skills