quark-torch-llm-eval

Files (6)

• SKILL.md
• backends.md
• eval-frameworks.md
• model-inspection.md
• templates.md
• troubleshooting.md

SKILL.md (first lines)

# LLM Evaluation on AMD ROCm

End-to-end LLM accuracy evaluation on AMD ROCm — from Docker container to final accuracy table.

## When to use

Trigger this skill when the user asks to:

- Evaluate, benchmark, or test a model's accuracy
- Run gsm8k / mmlu / mmlu_pro / aime / math_500 / gpqa / hellaswag / arc
- Deploy a model with vLLM, SGLang, or ATOM and measure accuracy
- Compare quantized model quality against a reference
- Verify a newly quantized model ("does this mxfp4 model lose accuracy?")

## Inputs

| Parameter | Required | Description |
|-----------|----------|-------------|
| `model_path` or `hf_id` | **Yes** | Local path to model weights, or HuggingFace model ID |
| `benchmark` | No | Benchmark to run (default: `gsm8k`) |
| `backend` | No | Serving backend: `vLLM` (default), `SGLang`, or `ATOM` |
| `image` | No | Docker image (default: `vllm/vllm-openai-rocm:latest`; overridden by model card / Recipes / user). Ignored when `runtime=host`. |
| `runtime` | No | `docker` (default) or `host`. Auto-detected in Phase 0.5: `docker` when the daemon is reachable, else `host` when a native `vllm` / `sglang` is importable. ATOM is docker-only. Stop and report if neither path is available. |

## Outputs

`$EVAL_STATE_DIR/eval_report.md` — see `templates.md` §2 for the full skeleton, formatting rules, and the example used as the canonical shape.

## Design Principles

1. **Trust the model card** — launch commands from model card / Recipes / Cookbook are author-verified. Use them as-is, only adjusting TP and `HIP_VISIBLE_DEVICES`. Do not preemptively stack flags.
2. **Reactive error handling** — `troubleshooting.md` is a diagnostic guide, not a prevention checklist. Launch first, consult on failure.

## Interaction Flow

### Unified confirmation gate

After Phase 0 + 0.5 + 1 finish collecting all information, present a **single confirmation summary** to the user before any container or server operations begin. Use the template in `templates.md` §1 — it defines all required fields, command format, and display rules.

Wait for explicit user approval before executing any of these commands. This avoids piecemeal confirmations and lets the user review the full pipeline at once.

**After approval**, the following are pre-approved by the unified gate — execute without re-asking:

- `docker run` / `docker exec` / `docker pull` of the planned container/image
- The exact `vllm serve` / `python -m sglang.launch_server` / `python -m atom.entrypoints.openai_server` command shown in the plan (with the planned GPU/TP/port)
- The exact `lm_eval` / `lighteval` / `evalscope` command shown in the plan
- `pip install` of frameworks listed in `eval-frameworks.md` §Installation — **docker runtime only** (inside the planned container). In host runtime, pip install is never pre-approved; see Phase 0.5 host-mode constraints.

Each of the following **still requires per-command confirmation**, even after the unified gate:

- Any edit to model `config.json` or backend source code (show full diff first)
- `kill -9` / `sudo` / `rm -rf` / `mv` outside `$EVAL_STATE_DIR` and outputs
- Any `vllm serve` / eval command that **deviates** from the plan (different flags, different port, retry with new args)
- `docker stop` / `docker rm` (Phase 7 cleanup — always ask)

Read-only probes (`ls`, `cat`, `du`, `rocm-smi`, `docker ps`, `docker images`, `curl -sI`, `curl -fsSL`, Read) do **not** need confirmation.

### Never kill other users' processes

- Never use `pkill -f <pattern>` or `ps | grep | awk | kill` broad matching
- Verify any PID is alive and belongs to the expected backend via `kill -0 <pid>` + `ps -fp <pid>` (check `USER` and `CMD` columns) before sending a signal — PIDs are recycled
- Send `SIGTERM` first, wait up to 30 s, then `SIGKILL` only if still alive

### HuggingFace fetch rules

All HuggingFace HTTP requests follow `model-inspection.md` §2 — `curl -fsSL`, `${HF_ENDPOINT:-https://huggingface.co}`, token from `~/.hf_token` then `$HF_TOKEN`. Read that section before the first HF request in any phase.

### Phase 0 — Auto-discover

**Prerequisite: user must provide `model_path` (local path) or `hf_id` (HuggingFace ID).**

- **local path** → verify it exists (`ls <model_path>/config.json`) → proceed
- **hf_id only** → fetch `config.json` and `tokenizer_config.json` from HF API (use the HuggingFace fetch rules above), then continue to Phase 0.5 → 1. Before Phase 2, ask: **A** "I have a local copy" or **B** "Download from HuggingFace"
- **model name only** (e.g., "llama 7b") → search HuggingFace, list candidates, let user choose → handle as hf_id
- **nothing** → ask for model_path or hf_id

Once a local path is confirmed, read `model-inspection.md` and follow its rules (infer hf_id, parse config.json, detect model type, estimate size, compute TP candidates). Write findings to `$EVAL_STATE_DIR/current-eval.yaml`.

### Phase 0.5 — Environment preflight (fail-fast)

Before Phase 1, run a 60-second environment probe to fail fast:

```bash
which curl python3
# Probe both runtimes; do NOT fail if docker is missing — host mode covers that.
docker --version >/dev/null 2>&1 && docker info >/dev/null 2>&1 \
  && echo "docker_available=true" || echo "docker_available=false"
python3 -c "import vllm; print('vllm', vllm.__version__)" 2>/dev/null \
  && echo "host_vllm_available=true" || echo "host_vllm_available=false"
python3 -c "import sglang; print('sglang', sglang.__version__)" 2>/dev/null \
  && echo "host_sglang_available=true" || echo "host_sglang_available=false"
```

**Runtime selection (record `runtime:` in `current-eval.yaml`):**

1. If the caller passed `runtime` explicitly (`docker` or `host`), honour it; validate the matching tool exists.
2. Else if `docker_available=true`, use `docker` (default — production path).
3. Else if `backend=vLLM` and `host_vllm_available=true`, or `backend=SGLang` and `host_sglang_available=true`, fall back to `runtime=host` and **report the fallback in the unified confirmation**.
4. Else stop and report: docker is not available, and the requested backend is not importable on the host either. List which of (docker, vllm, sglang) is missing and instruct the user to install one. **ATOM is docker-only** — if `backend=ATOM` and docker is missing, stop here regardless of host imports.

Host-mode constraints (announce in the unified confirmation):

- `image` is ignored; the host's currently-installed framework version is what runs.
- All later phases that show `docker exec ${CONTAINER} <cmd>` become `<cmd>` directly on the host (the "Host runtime variant" call-out under each phase below).
- `pip install` of frameworks is NOT performed — the host environment is used as-is. Missing a benchmark framework (e.g., `lm_eval`) → stop and ask the user to install it explicitly; do not pip install on the host without consent.

**Session path setup (first thing in this phase):** generate a unique, per-session state directory to avoid collisions between concurrent evals.

`bash` shell state does NOT persist across separate `Bash` tool calls — `EVAL_USER` / `EVAL_TS` exported in one call are gone in the next. The path must therefore be **resolved once and persisted to disk**, not re-exported each phase.

```bash
EVAL_USER=$(whoami)
EVAL_TS=$(date +%Y%m%d_%H%M%S)
EVAL_STATE_DIR="/tmp/eval-state/${EVAL_USER}/${EVAL_TS}"
mkdir -p "${EVAL_STATE_DIR}"
echo "eval_state_dir: ${EVAL_STATE_DIR}" > "${EVAL_STATE_DIR}/current-eval.yaml"
echo "${EVAL_STATE_DIR}" > /tmp/eval-state/.last-session-path-${EVAL_USER}
```

From amd/Quark, MIT.

More from amd/Quark