quark-torch-result-validator

Files (3)

• SKILL.md
• quant_validation.py
• run_validation.py

SKILL.md (first lines)

# quark-torch-result-validator

## Purpose

Run four lightweight checks on a completed Quark export or file2file output. All checks read only
safetensors headers or small metadata files — no full tensor payload is loaded except for the bounded
MD5 spot-check. Results feed into a structured `validation_report.md`.

## Runtime Assumptions

All scripts (`quant_validation.py`, `run_validation.py`) live in the same directory as this `SKILL.md`,
under `skills/_legacy_impl/l1-atomic/torch/quark-torch-result-validator/`.

Resolve `SKILL_DIR` from the repo root before running any command:

```bash
SKILL_DIR=skills/_legacy_impl/l1-atomic/torch/quark-torch-result-validator
```

`run_validation.py` writes JSON to **stdout**; `quant_validation.py` diagnostics go to **stderr**.
Never treat stderr as structured output.

## Contracts

- Input: `session_context.json`, `quant_plan.json` (for exclude rules and model paths)
- Output: `validation_report.md` (structured per the report template below)
- Schemas: `shared/contracts/validation_report.schema.json`

## Inputs

| Field | Source | Required |
|-------|--------|----------|
| `source_model_dir` | user or `session_context` | Yes |
| `quantized_model_dir` | user or `run_manifest` | Yes |
| `quant_config` | `quant_plan.json` or user-supplied JSON | For step 2 only |

## Outputs

`validation_report.md` with one section per executed step. Unexecuted steps are marked **skipped**.

## Interaction Flow

1. Confirm `SKILL_DIR`, `source_model_dir`, and `quantized_model_dir` are available.
2. Run self-test to verify scripts are intact: `python3 "$SKILL_DIR/run_validation.py" self-test`
3. Execute steps in cheap-to-expensive order (4 → 1 → 3 → 2).
4. Collect JSON from stdout for each step; write `validation_report.md`.
5. Surface any `ok: false` steps with their `errors` and `mismatches`.

## Four Validation Steps

| Order | Function | CLI subcommand | Purpose |
|-------|----------|----------------|---------|
| 1 | `check_auxiliary_files_copied` | `auxiliary` | Compare non-weight auxiliary files between source and quantized dirs |
| 2 | `check_non_quantized_tensors_md5_unchanged` | `md5` | MD5 spot-check payload bytes of exclude-listed tensors |
| 3 | `check_config_json_equal_except_quantization` | `config` | Deep compare `config.json` after stripping quantization keys |
| 4 | `get_fuzzy_tensor_names` | `fuzzy` | Summarize canonical header patterns and `dtype_counts` |

Run in cost order: **4 → 1 → 3 → 2**.

## Agent Execution Contract

### Self-Test

```bash
SKILL_DIR=skills/_legacy_impl/l1-atomic/torch/quark-torch-result-validator
python3 "$SKILL_DIR/run_validation.py" self-test
```

Exits `0` and prints `exported symbols self-test (__all__): ok` on success.

### CLI Commands

```bash
SKILL_DIR=skills/_legacy_impl/l1-atomic/torch/quark-torch-result-validator

# 4. get_fuzzy_tensor_names (cheapest — header only)
python3 "$SKILL_DIR/run_validation.py" fuzzy \
  --model-path ./quantized-model

# 1. check_auxiliary_files_copied
python3 "$SKILL_DIR/run_validation.py" auxiliary \
  --source-model-dir ./original-model --quantized-model-dir ./quantized-model \
  --ignore 'README*'

# 3. check_config_json_equal_except_quantization
python3 "$SKILL_DIR/run_validation.py" config \
  --original-config ./original-model/config.json \
  --quantized-config ./quantized-model/config.json

# 2. check_non_quantized_tensors_md5_unchanged (most expensive)
python3 "$SKILL_DIR/run_validation.py" md5 \
  --source-model-dir ./original-model --output-model-dir ./quantized-model \
  --quant-config '{"exclude":["*.lm_head.weight"],"max_samples":50}'
```

For `md5`, `--quant-config` accepts a JSON string or a path to a JSON file.
If `SKILL_DIR` or model paths cannot be resolved, mark the affected step **skipped**.

## Public API (for direct Python import)

```python
from quant_validation import (
    check_auxiliary_files_copied,
    check_non_quantized_tensors_md5_unchanged,
    check_config_json_equal_except_quantization,
    get_fuzzy_tensor_names,
)
```

All four functions are in `__all__`. Other public-named helpers are internal utility surface.

## Recovery

| Failure | Recovery |
|---------|----------|
| Self-test exits non-zero | Report script integrity failure; do not run further steps |
| `source_model_dir` missing | Mark steps 1, 2, 3 as skipped; run step 4 on quantized dir only |
| No safetensors files found | Mark steps 2 and 4 as skipped |
| `quant_config` missing exclude rules | Mark step 2 as skipped |

From amd/Quark, MIT.

More from amd/Quark