quark-torch-llm-ptq-workflow

Files (2)

• SKILL.md
• example-fp8-qwen3-8b.md

SKILL.md (first lines)

# quark-torch-llm-ptq-workflow

## Purpose

Chain the full PTQ path — model intake → quantization planning → manifest generation → confirmed execution — while keeping the user informed at each checkpoint. This workflow orchestrates the atomic skills so the user does not have to manually chain them. It stops at the quantized output; for a run that also validates and evaluates the result, use the `quark-torch-llm-ptq-eval` recipe.

## Inputs

- Model path (HuggingFace ID or local directory)
- User goal (target precision, hardware, accuracy)
- Output directory for the quantized model
- `session_context.json` for user goal and constraints
- `env_context.json` for hardware facts
- `workspace_context.json` for validated paths
- `pytorch_install_result.json` and `quark_install_result.json` to confirm runtime is ready

## Outputs: run_manifest.yaml

Records the executed command and config. Side artifacts: quantized model files written to the user's output directory. The manifest is built in Step 3 above.

Schema: [`run_manifest.schema.json`](../../../shared/contracts/run_manifest.schema.json)

## Interaction Flow

1. **Intake** → `quark-torch-model-intake` → `model_analysis.json`
2. **Plan** → `quark-torch-quant-plan` → `quant_plan.json`
3. **Confirm** → `run_manifest.yaml` (stop for user approval)
4. **Execute** → run the confirmed command, write quantized model

## CRITICAL RULES

1. **NEVER run `quantize_quark.py` directly.** Always go through the 4 steps below.
2. **NEVER skip a step.** Even if the user provides all details upfront, execute each step in order.
3. **STOP at every checkpoint** and wait for user confirmation before continuing.
4. **Show concrete output** at each step — tables, JSON, commands — not just prose descriptions.
5. **NEVER modify Quark's own source code or examples.** The Quark repo (`quark/`, `examples/`, `tools/`, `docs/`, `tests/`) is read-only from this workflow's perspective. See *Upstream Quark Code is Read-Only* below.

## Upstream Quark Code is Read-Only

The Quark repository (`quark/`, `examples/`, `tools/`, `docs/`, `tests/`, `pyproject.toml`, `requirements.txt`) may be read freely but never modified. This includes "just to make the script accept my flag" patches to `examples/torch/language_modeling/llm_ptq/quantize_quark.py` — they break reproducibility against a clean Quark install.

**When the shipped example does not cover the user's needs**, write a fresh standalone script in the user's working directory (or `/tmp/`) that imports from `quark`:

```python
# user_workspace/my_custom_ptq.py — NOT inside the Quark repo
from quark.torch import ModelQuantizer
from quark.torch.quantization.config.config import Config
# ... user-specific logic ...
```

Reference *that* script in `run_manifest.yaml`. The shipped `quantize_quark.py` stays untouched; the run remains reproducible against any Quark version. If the user actually needs an upstream Quark change, surface it as a contribution — do not silently patch their local checkout.

## Required Artifact Flow

```text
Step 1 (Intake)   ──► model_analysis.json
Step 2 (Plan)     ──► quant_plan.json
Step 3 (Manifest) ──► run_manifest.yaml      (contains the exact command)
Step 4 (Execute)  ──► quantized model output (only after user says yes)
```

---

## Step 1: Model Intake

**Goal**: Produce `model_analysis.json`.

### Actions

1. Locate the Quark PTQ script (`quantize_quark.py`) under `<Quark repo>/examples/torch/language_modeling/llm_ptq/` — `find / -name quantize_quark.py -path "*/llm_ptq/*" 2>/dev/null | head -1` if the path is unknown. Record it for Step 3.

2. Call `quark-torch-model-intake` with the model path. It handles config parsing (no weight load), supported-template matching, and risk identification (MoE, >70B, transformers version constraints) and emits `model_analysis.json`.

### Output to Show User

Render the summary from `model_analysis.json`:

```text
Model Analysis:
  Model path:       Qwen/Qwen3-8B
  Model type:       qwen3
  Hidden layers:    36
  Linear layers:    ~224 (quantization targets)
  MoE:              No
  Exclude defaults: [lm_head]
  Risks:            <list>
  Compatibility:    OK
```

### >>> CHECKPOINT 1: Confirm model analysis is correct before continuing

---

## Step 2: Quantization Plan

**Goal**: Build `quant_plan.json` from the model analysis and user's stated preferences.

### Actions

1. **Determine the scheme.** If the user stated a scheme (e.g., "FP8"), use it. Otherwise, recommend based on their priority:

   | Priority | Recommended Scheme | Algorithm |
   |----------|--------------------|-----------|
   | Best accuracy | `fp8` or `ptpc_fp8` | `smoothquant` (optional) |
   | Smallest model | `int4_wo_32` | `awq` or `gptq` |
   | CPU deployment | `int8` | none |
   | AMD MI300X/MI355X | `fp8` or `amdfp4` | none |
   | NVIDIA H100 | `fp8` | none |
   | GGUF export | `uint4_wo_32` | `awq` |

2. **Fill the decision table.** Show ALL decisions with defaults:

   | Decision | Value | Reason |
   |----------|-------|--------|
   | `global_scheme` | `fp8` | User requested FP8 |
   | `kv_cache_scheme` | `fp8` | Recommended for FP8 inference |
   | `exclude_layers` | `["lm_head"]` | Standard — lm_head stays full precision |
   | `layer_quant_config` | `{}` | No per-pattern overrides (or e.g. `{"*self_attn*": "fp8"}` if the user asked to quantize attention with a non-global scheme) |
   | `algorithm` | `null` | RTN baseline (fastest) |

From amd/Quark, MIT.

More from amd/Quark