quark-torch-model-intake

Files (1)

• SKILL.md

SKILL.md (first lines)

# quark-torch-model-intake

## Purpose

Validate the target model and extract the structural facts that `quark-torch-quant-plan` needs to make correct quantization decisions. This step exists because different model architectures have different quantization requirements — MoE models need expert module replacement, some models require `trust_remote_code`, and certain architectures have known compatibility issues with specific transformers versions.

## Inputs

- Model path (HuggingFace ID or local directory)
- `env_context.json` for Python and accelerator constraints
- `workspace_context.json` for the validated model path

## Outputs: model_analysis.json

Captures architecture facts, quantizable layer count, transformer compatibility, and risks.

Schema: [`model_analysis.schema.json`](../../../shared/contracts/model_analysis.schema.json)

```json
{
  "analysis_status": "complete",
  "model": {
    "model_path": "Qwen/Qwen3-8B",
    "model_type": "qwen3",
    "trust_remote_code": false,
    "transformers_version_required": ">=4.48.0,<5.3",
    "loading_class": "AutoModelForCausalLM",
    "is_moe": false,
    "num_hidden_layers": 36,
    "hidden_size": 4096,
    "estimated_size_gb": 16.0
  },
  "quantization_targets": {
    "linear_layer_count": 224,
    "has_non_linear_experts": false,
    "exclude_defaults": ["lm_head"],
    "needs_moe_preparation": false
  },
  "risks": [
    {
      "severity": "low",
      "message": "Model size fits in single GPU with 24GB+ VRAM for FP8/INT8 schemes.",
      "recovery_hint": "Use --multi_gpu for INT4 with large calibration datasets if OOM occurs."
    }
  ]
}
```

## Supported Model Architectures

Quark has built-in templates for 36 model families:

| Category | Models |
|----------|--------|
| **Standard LLMs** | llama, mistral, opt, phi, phi3, qwen, qwen2, gptj, cohere, olmo |
| **Advanced LLMs** | qwen3, qwen3_next, deepseek, deepseek_v2, deepseek_v3, deepseek_v32 |
| **MoE Models** | mixtral, dbrx, llama4, qwen2_moe, qwen3_moe, qwen3_5_moe, gpt_oss, granitemoehybrid, glm4_moe |
| **Vision-Language** | mllama, deepseek_vl_v2, qwen3_vl_moe |
| **Other** | chatglm, gemma2, gemma3, gemma3_text, grok-1, instella, kimi_k25, minimax_m2 |

Models not in this list may still work if they follow standard HuggingFace `AutoModelForCausalLM` patterns, but need extra attention.

## What to Extract

### From config.json

- `model_type` — must match a Quark template name (e.g., `"llama"`, `"qwen3"`, `"mistral"`)
- `num_hidden_layers` — determines the number of quantizable linear layers
- `hidden_size`, `intermediate_size` — affects memory estimates
- `num_attention_heads`, `num_key_value_heads` — relevant for KV cache quantization
- Architecture-specific fields (e.g., `num_experts` for MoE models)

### Quantization Targets

- Count total linear layers (these are what gets quantized)
- Identify default exclusions — `lm_head` is almost always excluded
- For MoE models, note that expert modules need special preparation via `prepare_for_moe_quant()`

### Transformer Compatibility

Some models require specific minimum transformers versions:

- `llama4` → transformers >= 4.51.0
- `gpt_oss`, `granitemoehybrid` → transformers >= 4.55.1
- `qwen3_vl_moe` → transformers >= 4.57.0
- `qwen3_5_moe` → transformers >= 5.2.0
- General requirement for LLM PTQ: `transformers < 5.3`

### Special Loading Requirements

- `deepseek_vl_v2` → uses `AutoModel` instead of `AutoModelForCausalLM`
- `mllama` → uses `MllamaForConditionalGeneration`
- `gpt_oss` → needs `Mxfp4Config(dequantize=True)` for loading
- Models with custom code → need `trust_remote_code=True`

### Risks

Flag anything that could cause failures downstream:

- Model type not in Quark's template list
- Transformer version incompatibility
- Very large models that may need `--multi_gpu` or `--file2file_quantization`
- MoE models that need module replacement

## Concrete Actions

### Action 1: Read model config (NEVER load full weights)

```bash
python3 -c "
from transformers import AutoConfig
import json
config = AutoConfig.from_pretrained('<MODEL_PATH>', trust_remote_code=True)
info = {
    'model_type': config.model_type,
    'num_hidden_layers': config.num_hidden_layers,
    'hidden_size': config.hidden_size,
    'intermediate_size': getattr(config, 'intermediate_size', None),
    'num_attention_heads': config.num_attention_heads,

From amd/Quark, MIT.

More from amd/Quark