quark-torch-llm-ptq

Files (12)

• .federated.json
• SKILL.md
• evals.json · evals/
• model_analysis.schema.json · references/contracts/
• quant_plan.schema.json · references/contracts/
• run_manifest.schema.json · references/contracts/
• environment.md · references/
• example-fp8-qwen3-8b.md · references/
• model-intake.md · references/
• quant-plan.md · references/
• troubleshooting.md · references/
• skill-card.md

SKILL.md (first lines)

# Quark Torch PTQ

## Purpose

Take a PyTorch / Hugging Face LLM from model identification through confirmed
AMD Quark PTQ. Perform intake, planning, manifest generation, execution, and
output verification as one self-contained workflow.

This skill stops at the quantized model. It does not accept `.onnx` model input,
train or fine-tune a model, or modify Quark package/source files.

## Prerequisites

- Python 3.11 to 3.13, accelerator-matched PyTorch 2.2 or newer, `amd-quark[cli]`, and `datasets`.
- The required ROCm version and GPU architecture depend on the PyTorch build and selected quantization scheme. Record the actual runtime, GPU architecture (`gfx...` from `gcnArchName` on AMD), host kernel, and driver, and verify support for the confirmed plan.
- No container image is required. If running in a container, record its image name or digest when available and verify GPU device access.
- Inspect and preserve `HIP_VISIBLE_DEVICES`, `CUDA_VISIBLE_DEVICES`, `HSA_OVERRIDE_GFX_VERSION`, `PYTORCH_ROCM_ARCH`, and `PYTORCH_HIP_ALLOC_CONF`. Include any required changes to device visibility, architecture, or memory allocation in the confirmed plan.

## Inputs

- Model source: a Hugging Face repository ID or local model directory.
- Output directory.
- Quantization intent: requested precision/scheme, target hardware, accuracy
  priority, and optional calibration settings.
- Optional environment facts: Python, PyTorch, accelerator, available memory,
  installed `amd-quark`, and Transformers versions.

Do not require pre-existing workflow artifacts. Create all artifacts in the
user's working directory by following this skill's local references:

- [`references/model-intake.md`](references/model-intake.md)
- [`references/quant-plan.md`](references/quant-plan.md)
- [`references/environment.md`](references/environment.md)
- [`references/troubleshooting.md`](references/troubleshooting.md)

## Outputs

Produce these three artifacts before or during execution:

1. `model_analysis.json`, validated against
   [`references/contracts/model_analysis.schema.json`](references/contracts/model_analysis.schema.json).
2. `quant_plan.json`, validated against
   [`references/contracts/quant_plan.schema.json`](references/contracts/quant_plan.schema.json).
3. `run_manifest.yaml`, validated against
   [`references/contracts/run_manifest.schema.json`](references/contracts/run_manifest.schema.json).

The quantized model and its configuration/tokenizer files are written under the
confirmed output directory. Record actual files and the final status in the
manifest.

## Interaction Flow

Always complete the following four steps in order. Show concrete facts,
artifacts, and commands. Stop at every checkpoint and wait for the user.

### Step 1 — Model intake

1. Confirm whether the model source is local or remote. For a local source,
   resolve it to an absolute path and verify that the directory and
   `config.json` exist. For a remote source, preserve the repository ID.
2. Follow `references/model-intake.md`. Read configuration only; do not load
   model weights during intake.
3. Determine `model_type`, architecture/loading hints, hidden-layer facts,
   multimodal or MoE signals, default exclusions, compatibility risks, and a
   defensible estimate of quantizable linear layers.
4. Write schema-valid `model_analysis.json` and show its summary.

#### Checkpoint 1 — Confirm the model analysis

Ask the user to confirm or correct the model analysis.

Do not plan quantization until the user confirms.

### Step 2 — Quantization plan

1. Follow `references/quant-plan.md` and use the confirmed analysis plus the
   user's priorities.
2. Select and explain the global scheme, optional KV-cache scheme, per-pattern
   overrides, exclusions, algorithms, and calibration data.
3. Treat scheme, algorithm, and model-template support as version-dependent.
   Use the current installed Quark API or `quark-cli torch-llm-ptq --help` when
   a choice needs verification; do not rely on historical list sizes.
4. Show a decision table, write schema-valid `quant_plan.json`, and set
   `requires_confirmation: true` until approved.

#### Checkpoint 2 — Confirm the quantization plan

Ask the user to confirm or adjust the complete plan.

After confirmation, update `requires_confirmation` to `false`. If the user
changes a decision, rewrite and revalidate the plan before continuing.

### Step 3 — Manifest and execution confirmation

Use the public `quark-cli torch-llm-ptq` command installed by
`amd-quark[cli]`. Do not import its implementation module directly or locate,
copy, generate, or patch another PTQ runner.

Build an argument-array-safe command equivalent to:

```bash
quark-cli torch-llm-ptq \
  --model_dir "<MODEL_OR_ABSOLUTE_LOCAL_PATH>" \
  --output_dir "<ABSOLUTE_OUTPUT_PATH>" \
  --quant_scheme "<SCHEME>" \
  --num_calib_data "<N>" \
  --seq_len "<LENGTH>" \
  --device cuda \
  --no_trust_remote_code
```

Add only confirmed options:

- `--dataset <NAME>` and `--batch_size <N>` when the plan changed them from the
  CLI defaults.
- `--kv_cache_dtype <SCHEME>` for confirmed KV-cache quantization.
- One `--layer_quant_scheme <PATTERN> <SCHEME>` per override.
- `--quant_algo <comma-separated-list>` when required.
- `--exclude_layers <patterns...>` only when overriding template defaults.

From amd/skills, MIT.

More from amd/skills