torch-integration-capability-evaluation

Files (14)

• marketplace.json · .claude-plugin/
• plugin.json · .claude-plugin/
• architecture-review-checklist.md · .github/prompts/
• torch-ice-review-agent.md · .github/prompts/
• torch-ice-review-agent.mjs · .github/scripts/
• torch-ice-review-agent.test.mjs · .github/scripts/
• torch-ice-review-agent.yml · .github/workflows/
• .gitignore
• LICENSE
• README.md
• SKILL.md
• crcr-l1-onboarding.md · crcr/
• EVAL.md · frameworks/pytorch/
• checklist.md · frameworks/pytorch/

SKILL.md (first lines)

# Check Integration Capability

You are an accelerator integration evaluator. Given a backend name or source
path, you evaluate its integration capability with **PyTorch**. You produce
scored readiness reports with concrete findings.

## Inputs

The user provides one of:
- A backend name (e.g., `ascend npu`, `habana gaudi`, `openreg`)
- A source path (e.g., `/home/user/torch_npu`)
- A GitHub URL

Optional flags:
- `--pytorch-version <version>` (e.g., `--pytorch-version 2.4.0`)
  Evaluate the backend against a specific PyTorch upstream version instead
  of the version the backend targets. Useful for checking compatibility
  with a newer or different PyTorch release. If omitted, the skill detects
  the PyTorch version from the backend's own dependency metadata, falling
  back to the latest stable PyTorch release.

If no input is provided, ask for one.

## Output Format -- MANDATORY

The final report **MUST** be a proper filled-in markdown checklist, not free-form
text or inline summaries. For each evaluation:

1. **Read the checklist template** from `frameworks/pytorch/`
   - `checklist.md` for PyTorch evaluation

2. **Copy the template** to `torch-ice-report/` as the working report file

3. **Fill every table row** in the copied markdown with:
   - `Points` column: 2 = fully implemented, 1 = partially implemented, 0 = not implemented, N/A = excluded
   - `Notes` column: concrete evidence (e.g., "Registered as 'npu' at backend.py:7", "Throws NotImplementedError", "23/30 ops pass")
   - `Priority` column: already pre-filled in the template (1-3 per row; 1=critical, 2=important, 3=nice-to-have)

4. **Fill the Readiness Score & Summary** (at the top of the document):
   - Row weight: `w_i = 1 / priority_i` (P1=1.0, P2=0.5, P3=0.333)
   - Row score: 2=fully implemented, 1=partially, 0=not implemented, N/A=excluded; max score per row = 2
   - Compute per-section: `section_pct = sum(score_i * w_i) / sum(max_i * w_i) * 100` (excluding N/A rows, where max_i=2)
   - Compute tier weight: `weight_r = 1 / level`
   - Compute overall readiness (weighted only): `(sum(section_pct * weight_r) / sum(weight_r)) * 100`
   - Fill the score table (sorted by level), overall readiness percentage, and executive summary. Report only the weighted percentage -- do not show an unweighted total.

5. **Append an Appendix** listing any discovered APIs not in the checklist

The output file must be a **complete, standalone markdown document** that renders
correctly and can be shared as-is. Never skip the markdown report in favor of
an inline text summary -- the filled checklist IS the deliverable.

## Template Selection

Before evaluation, determine which PyTorch template to use:

1. **Open-source backend** (source code available via local repo, GitHub, or cloneable):
   Use `frameworks/pytorch/checklist.md`
   This template uses source-code probing (grep, TorchTalk) and scored checklists.

2. **Private backend** (no public source, vendor-specific execution stack):
   Use `frameworks/pytorch/checklist_private.md`
   This template produces a narrative research document using public information
   (vendor docs, blogs, benchmarks, community evidence, runtime introspection).
   Covers accelerators that bypass standard PU1/Fork paths: AOT compilation,
   model conversion, lazy tensors, compile-only, or remote API backends.

Detection heuristic:
- Source code found (local, GitHub, cloneable)? -> open-source template
- Only pip package with no accessible source? -> private template
- Vendor uses model conversion + AOT compilation (not PU1 dispatch)? -> private template
- User explicitly specifies private/proprietary? -> private template

## Time Estimate

After the backend has been located and its integration path detected (open-source
vs. private, PrivateUse1 vs. Fork) but **before** the scored probing begins, print
a tentative estimate of how long generating this report will take for this specific
accelerator. Judge the range from the runtime drivers you just discovered:

- **Integration path** -- a private/narrative evaluation does web research and is
  slower than an open-source source-code probe.
- **Codebase size** -- more source to grep and trace takes longer.
- **Applicable sections** -- how many of the framework sections apply to this backend.

Present it as a rough range and state clearly that it is an approximation, e.g.:

```
Estimated report time: ~8-12 min (open-source, PrivateUse1, ~15 applicable sections).
This is a rough estimate, not a guarantee.
```

Record the wall-clock time at this step so the actual duration can be reported in
the final summary.

## Output Files

Create `torch-ice-report/` in the current project if it doesn't exist. Write:
- `torch-ice-report/torch_readiness_report_<backend>.md` -- open-source scored checklist
- `torch-ice-report/torch_readiness_research_<backend>.md` -- private backend narrative research
- Print summary to user at the end

---

## Framework Dispatch

Each framework lives under `frameworks/<name>/` with its own `EVAL.md` and
checklist templates. To add a new framework, create the directory and add an
entry to the dispatch table below. Only frameworks listed here are evaluated.

| Framework | Directory | When to evaluate | EVAL.md |
|-----------|-----------|-----------------|---------|
| PyTorch | `frameworks/pytorch/` | Always | `frameworks/pytorch/EVAL.md` |

For each framework in the table, read its `EVAL.md` and follow all phases.

---

## Final Output: Summary

From TorchedHat/torch-ice, Apache-2.0.

More from TorchedHat/torch-ice