quark-onnx-debug

Files (1)

• SKILL.md

SKILL.md (first lines)

# quark-onnx-debug

## Purpose

Convert ONNX-flow failures into a structured diagnostic report with the smallest safe recovery path.
Debugging Quark ONNX issues is tricky because errors can originate from many layers — Python
environment, the `onnx` package version, the installed `onnxruntime*` variant, custom-op
compilation, CUDA / ROCm / NPU EPs, calibration data plumbing, graph optimization, or the chosen
quantization config itself. This skill systematically narrows down the root cause.

## Inputs

- Error message and stack trace from a failing run
- The exact command or `ModelQuantizer` / `quantize_static` invocation that triggered it
- `env_context.json`, `onnx_install_result.json`, `quark_install_result.json` (optional, for
  environment and install state)
- The model path / size and (if relevant) the `QuantizationConfig` used

## Outputs: validation_report.md

Diagnostic report with root cause, evidence, and the smallest safe fix.

Schema: [`validation_report.schema.json`](../../../shared/contracts/validation_report.schema.json)

````markdown
# Debug Report

## Common Error Patterns

### Installation Errors

| Error | Likely Cause | Fix |
|-------|-------------|-----|
| `ModuleNotFoundError: No module named 'quark'` | Quark not installed or wrong Python env | `pip install amd-quark` or activate correct conda env |
| `ModuleNotFoundError: No module named 'onnxruntime'` | ONNX Runtime not installed | Hand off to `quark-onnx-install` |
| `ModuleNotFoundError: No module named 'onnx'` | `onnx` package not installed | `pip install "onnx>=1.21.0,<=1.22.0"` |
| `ImportError: quark.onnx.operators.custom_ops` / compile failure | Missing C++ compiler (Linux `g++`, Windows VS 2022) or `ROCM_PATH`/`CUDA_HOME` unset for GPU build | `apt install g++` (Linux), install VS 2022 (Windows), `export ROCM_PATH=/opt/rocm` or `export CUDA_HOME=/usr/local/cuda` |
| Both `onnxruntime` and `onnxruntime-gpu` (or `_rocm`) installed | Variant collision — wrong EP loads | `pip uninstall -y onnxruntime onnxruntime-gpu onnxruntime_rocm`, then reinstall the single correct variant via `quark-onnx-install` |
| `onnx` schema / `opset` errors at import | `onnx` outside `>=1.16.0,<=1.19.0` | Pin within the supported range |
| `onnxruntime` ABI / symbol errors when loading custom ops | ORT version outside `>=1.22.2,<=1.24.2` (custom-ops built against a different ABI) | Reinstall ORT within the supported range, then re-trigger custom-ops compile |

### ONNX Runtime / Execution Provider Errors

| Error | Likely Cause | Fix |
|-------|-------------|-----|
| `get_available_providers()` lacks `CUDAExecutionProvider` | CPU `onnxruntime` installed on a CUDA box | Reinstall `onnxruntime-gpu` via `quark-onnx-install` |
| `get_available_providers()` lacks `ROCMExecutionProvider` on ROCm 6.x | Wrong variant (need `onnxruntime_rocm` from AMD Artifactory) | Reinstall `onnxruntime_rocm` via `quark-onnx-install` |
| `ROCMExecutionProvider` missing on ROCm 7.x | **By design** — ROCm 7.x falls back to CPU `onnxruntime` per `tools/ci/install_onnxruntime.sh` (build incompatibility) | Document the CPU-only fallback; do not attempt a ROCm wheel install |
| Silent CPU fallback (no GPU utilization during calibration) | EP not requested or unavailable | Pass `execution_providers=['CUDAExecutionProvider']` / `['ROCMExecutionProvider']` explicitly; verify with `get_available_providers()` |
| `Failed to load library libonnxruntime_providers_cuda.so` | CUDA toolkit version mismatch with `onnxruntime-gpu` build | Match `onnxruntime-gpu` to the system CUDA major (CUDA 11 → Azure DevOps index; CUDA 12/13 → pypi default) |
| `RuntimeError: ... onnxruntime::rocm::...` | ROCm driver / library mismatch | Verify `rocm-smi` matches the wheel's ROCm version |

### Custom Op Errors

| Error | Likely Cause | Fix |
|-------|-------------|-----|
| `RuntimeError: Failed to load custom op library` | Custom-ops library was not compiled, or compiled against a different ORT version | Re-run `python -c "import quark.onnx.operators.custom_ops"` and inspect the compile output |
| `Op (BFPQuantizeDequantize) ... is not a registered function/op` | The session was created without registering Quark's custom-op library | Use `quark.onnx.ModelQuantizer` / `quantize_static`, or pass the custom-op `.so`/`.dll` via `SessionOptions.register_custom_ops_library()` |
| Custom-op symbol-not-found on GPU but works on CPU | GPU kernel of the custom op was not built (missing `ROCM_PATH` / `CUDA_HOME` at compile time) | Set the env var, delete the cached `.so`/`.dll`, re-import to recompile |

### Model Loading / Shape Errors

| Error | Likely Cause | Fix |
|-------|-------------|-----|
| `FileNotFoundError: Input model file ... does not exist.` (`api.py:103`) | Wrong `model_input` path | Check absolute path to `model.onnx` |
| `onnx.onnx_cpp2py_export.checker.ValidationError` | Model fails ONNX schema check | Run `onnx.checker.check_model()` to localize; rebuild the model with a compatible opset |
| `Message ... exceeds 2GB` / ProtoBuf size error | Model > 2 GB without external data | Save with `save_as_external_data=True`; pass `use_external_data_format=True` to the quantizer |
| `external data file not found` | `.onnx` moved but `.onnx_data` / weight blobs left behind | Move the model directory as a whole, or re-export with external data adjacent |
| Shape-inference failure | Incomplete shapes in the model | Run `quark.onnx.tools.fix_shapes` / `onnx.shape_inference.infer_shapes` before quantization |

### Calibration Errors

| Error | Likely Cause | Fix |
|-------|-------------|-----|
| `ValueError: The data reader should implement the '__len__' method to provide the data size.` (`calibrators.py:239`) | Custom `CalibrationDataReader` missing `__len__` | Implement `__len__` returning sample count |
| `ValueError: No data is collected.` (`calibrators.py:253`) | Data reader yielded zero batches before exhaustion | Check reader's `get_next()` returns at least one batch |
| `TypeError: compute_data must return a TensorsData not <...>` (`calibrators.py:259`) | Custom calibrator subclass returned wrong type | Return `onnxruntime.quantization.calibrate.TensorsData` |
| `ValueError: No collector created and can't generate calibration data.` | `collect_data()` never called before `compute_data()` | Call `collect_data()` first, or use `ModelQuantizer.quantize_model()` which handles the order |
| `ValueError: Invalid averaging constant, which should not be < 0 or > 1.` (`calibrators.py:563`) | `moving_average=True` with out-of-range constant | Use a value in `[0, 1]`, typically `0.01` |
| `ValueError: Unsupported calibration method` (`calibrators.py:1237`) | Typo or unsupported `CalibrationMethod` | Check `CalibrationMethod` enum in `quark.onnx.quantization.config.config` |
| OOM during calibration | Activation cache too large | Set `optimize_mem=True` (disk cache) and/or `optimize_disk=True`; reduce `worker_num` if memory bound; reduce calibration sample count |

### Quantization Config Errors

| Error | Likely Cause | Fix |
|-------|-------------|-----|
| `ValueError: Only ExtendedQuantFormat.QDQ supports wide bits quantization types.` (`input_check.py:67`) | INT16/UINT16/INT32 with `QuantFormat.QOperator` | Switch to `ExtendedQuantFormat.QDQ` |
| `ValueError: Fast finetune does not support int4 or uint4.` (`input_check.py:118`) | AdaRound/AdaQuant + INT4 weights | Disable fast-finetune for INT4, or use GPTQ instead |
| `ValueError: Invalid quant overrides ... for tensor ...` (`input_check.py:86`) | Tensor name in `extra_options['QuantOverrides']` not in the graph | Verify tensor name with `onnx.load(...).graph` |
| `ValueError: The per-channel quant override ... can not be applied on <quant_type>` (`input_check.py:94`) | Per-channel override on a quant type that doesn't support it (e.g., FP types, block formats) | Remove per-channel from that override |
| `ValueError: For the crypto mode, the input model should be in onnx.ModelProto format.` (`input_check.py:137`) | `crypto_mode=True` with a file path | Load model first: `model = onnx.load(path)` |
| `ValueError: quantization config must be one of Config and QConfig.` (`api.py:73`) | Passed a dict or wrong type | Build a `quark.onnx.quantization.config.Config` or `QConfig` |
| `ValueError: Unexpected config name: <X>` (`custom_config.py:993`) | Typo in `get_default_config(<X>)` preset name | Pick from `DefaultConfigMapping` (XINT8, S8S8_AAWS, A8W8, A16W8, BF16, BFP16, MX4/6/9, MXFP8/6/4, …) |
| `nodes_to_quantize` ignored | Node names don't match graph after pre-processing (NCHW→NHWC, BN folding) | Run pre-processing first, dump the post-preprocess graph, then re-select node names |

### Algorithm Errors

| Error | Likely Cause | Fix |
|-------|-------------|-----|
| AdaRound / AdaQuant device mismatch | `optim_device='cuda:0'` but no `CUDAExecutionProvider` available | Either set `optim_device='cpu'` or fix the ORT install via `quark-onnx-install` |
| AdaRound / AdaQuant divergence (loss → NaN/Inf) | LR too high for the model, or activation outliers | Lower `learning_rate`, enable `CLE` pre-processing first |
| GPTQ shape mismatch | Group-size doesn't divide the weight dimension | Pick a `group_size` that divides the weight's reduction dim (commonly 32, 64, 128) |
| SmoothQuant alpha range | `SmoothAlpha` outside `[0, 1]` | Use a value in `[0.5, 0.85]` for LLMs |
| QuaRot rotation config invalid | Missing rotation pair definitions | Provide the rotation pairs in the algorithm config; see `quark/onnx/algorithm/quarot/` |
| `AutoSearchPro: 'model_input' can not be None.` (`auto_search_pro.py:139`) | Search invoked without a model | Pass the model path or `ModelProto` |
| `AutoSearchPro: 'calib_data_reader' can not be None.` (`auto_search_pro.py:142`) | Search invoked without calibration data | Provide a `CalibrationDataReader` |
| `AutoSearchPro: Unsupported search_algo` (`auto_search_pro.py:192`) | Sampler typo | Use `'TPE'` or `'Grid'` |

### NPU-Specific Errors (Ryzen AI / VAI)

| Error | Likely Cause | Fix |
|-------|-------------|-----|
| Power-of-2 scale validation failure | Non-PoF2 scale produced for an NPU target | Use a PoF2 calibrator (`PowOfTwoCalibrater` MinMSE / NonOverflow); confirm `enable_npu_cnn=True` |
| `enable_npu_cnn=True` but model has unsupported ops | NPU CNN backend doesn't support the op | Fold / replace via `quark.onnx.tools.*`; see `optimizations/optimize.py` |
| NCHW vs NHWC mismatch | NPU expects NHWC but model is NCHW | Run `quark.onnx.tools.convert_nchw_to_nhwc` before quantization |
| `enable_dpu` deprecated warning | Old API | Switch to `enable_npu_cnn=True` |

### Export / Post-quantization Errors

From amd/Quark, MIT.

More from amd/Quark