quark-torch-quant-plan

Files (1)

• SKILL.md

SKILL.md (first lines)

# quark-torch-quant-plan

## Purpose

Convert a model analysis plus the user's intent into a confirmed `quant_plan.json`. This skill makes the quantization decisions — which scheme, which algorithm, what to exclude — without generating scripts or executing PTQ. The plan is the contract between the user's intent and the execution step.

## Inputs

- `model_analysis.json` from quark-torch-model-intake
- `env_context.json` for accelerator-aware scheme recommendations
- User preferences (target precision, accuracy goal)

## Outputs: quant_plan.json

Records the chosen scheme, algorithm, layer overrides, calibration settings, and evaluation intent.

Schema: [`quant_plan.schema.json`](../../../shared/contracts/quant_plan.schema.json)

```json
{
  "model": {
    "model_type": "qwen3",
    "analysis_ref": "./model_analysis.json"
  },
  "global_scheme": "fp8",
  "kv_cache_scheme": "fp8",
  "exclude_layers": ["lm_head"],
  "layer_quant_config": {},
  "algorithm": null,
  "calibration": {
    "dataset": "pileval",
    "num_calib_data": 128,
    "seq_len": 512
  },
  "evaluation_intent": "smoke",
  "requires_confirmation": false
}
```

## Available Quantization Schemes (21 total)

### Weight-Only INT4 (best for deployment size reduction)

| Scheme | Description | Use Case |
|--------|-------------|----------|
| `int4_wo_32` | INT4, group size 32 | Highest accuracy among INT4 |
| `int4_wo_64` | INT4, group size 64 | Good balance |
| `int4_wo_128` | INT4, group size 128 | Smaller overhead |
| `int4_wo_per_channel` | INT4, per-channel | Least overhead |
| `uint4_wo_32/64/128/per_channel` | Unsigned INT4 variants | GGUF export compatibility |

### Weight+Activation INT8

| Scheme | Description | Use Case |
|--------|-------------|----------|
| `int8` | INT8 per-tensor for both W and A | CPU deployment, good accuracy |

### FP8 (best accuracy-size tradeoff for GPU inference)

| Scheme | Description | Use Case |
|--------|-------------|----------|
| `fp8` | FP8 E4M3 per-tensor | Standard GPU quantization |
| `ptpc_fp8` | Per-Token-Per-Channel FP8 | Higher accuracy, dynamic activation quantization |

### OCP Microscaling Formats

| Scheme | Description | Use Case |
|--------|-------------|----------|
| `mxfp4` | OCP MXFP4 | Aggressive compression |
| `mxfp6_e3m2` | OCP MXFP6 (E3M2) | Better range |
| `mxfp6_e2m3` | OCP MXFP6 (E2M3) | Better precision |
| `mxfp4_mxfp6_e2m3` | MXFP4 weights + MXFP6 activations | Mixed precision |
| `mxfp4_fp8` | MXFP4 weights + FP8 activations | Mixed precision |

### AMD-Specific

| Scheme | Description | Use Case |
|--------|-------------|----------|
| `amdfp4` | amdfp4, group size 16 | AMD MI300X optimized |
| `amdfp4_g32` | amdfp4, group size 32 | AMD MI300X, less overhead |

### Other

| Scheme | Description | Use Case |
|--------|-------------|----------|
| `nvfp4` | NVFP4: FP4 group_size=16 with FP8 E4M3 scale | NVIDIA Blackwell/Hopper |
| `mx6` | MX6 format | Experimental |
| `bfp16` | Block Floating Point 16-bit | Experimental |
| `int4_wa_64` | INT4 weights + activations, group 64 | Research |

## Available Algorithms (7 primary)

| Algorithm | Compatible Schemes | Description |
|-----------|-------------------|-------------|
| `awq` | INT4/UINT4 weight-only | Activation-aware weight quantization — finds optimal per-channel scaling |
| `gptq` | INT4/UINT4 weight-only | Second-order weight optimization — often better than AWQ for small models |
| `smoothquant` | INT8, FP8 | Migrates quantization difficulty from activations to weights |
| `autosmoothquant` | INT8, FP8 | Automatic SmoothQuant with optimal alpha search |
| `rotation` | Various | Rotation-based optimization to equalize weight distribution |
| `gptaq` | INT4/UINT4 | GPTAQ variant combining GPTQ with activation quantization |
| `qronos` | Various | Custom algorithm for time-series-aware quantization |

Algorithms can be combined: `--quant_algo awq,smoothquant`

## KV Cache Quantization

- Only `fp8` is supported for KV cache (`--kv_cache_dtype fp8`)
- Adds `--min_kv_scale` option (default 0.0) to prevent extreme scale values
- `--kv_cache_post_rope` quantizes KV cache after RoPE (inside cache) instead of at k_proj/v_proj outputs — can improve accuracy for some models

## Decision Guide

Help the user choose based on their priorities:

**"I want the best accuracy"** → `fp8` or `ptpc_fp8`, optionally with `smoothquant`
**"I want the smallest model"** → `int4_wo_32` with `awq` or `gptq`
**"I need CPU deployment"** → `int8` (the only scheme that works well on CPU)
**"I need GGUF format"** → `uint4_wo_32` with `awq`, export as GGUF
**"I'm on AMD MI300X"** → `amdfp4` for best hardware utilization

From amd/Quark, MIT.

More from amd/Quark