magpie-kernel-evaluator

Files (10)

• .federated.json
• SKILL.md
• evals.json · evals/
• analyze_default.yaml · evals/files/analyze-simple-hip/examples/simple_hip_test/
• vector_add.hip · evals/files/analyze-simple-hip/examples/simple_hip_test/
• vector_add.hip · evals/files/compare-hip-variants/baseline/
• vector_add.hip · evals/files/compare-hip-variants/candidate/
• examples.md
• reference.md
• skill-card.md

SKILL.md (first lines)

# Magpie

Use Magpie for three connected jobs:

1. Benchmark an inference workload and collect throughput, latency, and traces.
2. Analyze or compare GPU kernels for correctness and performance.
3. Drive an optimization loop from a benchmark bottleneck to source, candidate kernels, and end-to-end validation.

Describe only capabilities supported by the checked-out Magpie version. Do not infer support for an unverified ROCm, GPU, framework, or experimental integration.

## Choose the workflow

| User goal | Workflow |
|---|---|
| Evaluate one implementation | `analyze` |
| Rank two or more implementations | `compare` |
| Measure model-serving performance | `benchmark` |
| Find expensive kernels in existing traces | standalone gap analysis |
| Explain a profiled inference workload | benchmark → TraceLens post-processing → stage/roofline review |
| Optimize an end-to-end workload | benchmark → TraceLens/gap analysis → source mapping → analyze/compare → re-benchmark |

Use a YAML config for reproducible or multi-step work. Use inline CLI arguments for small exploratory runs.

## Preflight

Installing this skill does not install the Magpie application (`magpie-eval`). For a preparation-only request, read the supplied fixtures and [reference.md](reference.md), write the requested plan or configuration, and identify commands that remain unverified. Do not install packages or execute a workload when the user forbids it; a missing Magpie installation does not prevent preparing a plan.

Before executing a Magpie workload:

1. Check `magpie --help` in the intended Python environment (Python 3.10+). If the command is missing, try `python -m Magpie --help` in that same environment; a working module entry point can be used instead of the CLI. `No module named Magpie` means the application is not installed in that environment. For errors inside an installed Magpie package, diagnose the reported import or dependency failure instead.
2. If Magpie is not installed, install it in the intended environment before continuing:

   ```bash
   python -m pip install git+https://github.com/AMD-AGI/Magpie.git
   ```

   For an existing Magpie source checkout, use `python -m pip install -e /path/to/Magpie` instead. The installed skill folder and a testcase workspace are not Magpie source checkouts. Re-run `magpie --help` or `python -m Magpie --help` to verify the installation; if it fails, report the error before attempting a workload.
3. Inspect the installed interface for the selected workflow (substitute `python -m Magpie` if using the module entry point):

   ```bash
   magpie --help
   magpie analyze --help
   magpie compare --help
   magpie benchmark --help
   magpie --gpu-info
   ```

4. Check required tools, model access, GPU visibility, writable output space, and container or Ray access as applicable. Installing the Python package does not install ROCm/CUDA, profilers, or model weights.
5. Read the repository compatibility matrix before making version claims. Treat ROCm or hardware not listed there as unverified until tested.
6. Record the exact config, model revision, image, environment variables, GPU allocation, and Magpie commit for benchmark comparisons.

## Analyze a kernel

Prefer a config when correctness or profiler settings matter:

```bash
magpie analyze --kernel-config path/to/kernel.yaml
```

For a quick single-kernel run:

```bash
magpie analyze path/to/kernel.hip --type hip --testcase "./run_test.sh"
```

Supported public kernel types are `hip`, `cuda`, `pytorch`, and `triton`. Use `--no-perf` only when the user wants correctness or execution validation without profiling.

Do not equate successful execution with numerical correctness. Supply a representative testcase whenever an optimized result will be accepted or rejected.

## Compare kernel variants

Compare at least two implementations and identify the baseline explicitly:

```bash
magpie compare --kernel-config path/to/compare.yaml
```

Keep inputs, tolerances, warmup, iteration count, GPU allocation, and profiler settings identical across candidates. Reject candidates that fail correctness before considering performance rankings.

For PyTorch without a testcase, Magpie's built-in check only verifies that each result is finite; it does not prove numerical equivalence between variants. Require a testcase for numerical validation.

## Benchmark inference

Prefer a checked-in benchmark config:

```bash
magpie benchmark --benchmark-config path/to/benchmark.yaml
```

The stable public CLI supports `vllm`, `sglang`, and `atom`. It supports direct `docker` and `local` run modes; use YAML configuration and the repository's Ray examples for distributed execution. Do not advertise integrations that exist only in internal enums or partial code paths as stable.

Enable profiling deliberately: profiler runs perturb latency and should not replace a clean baseline. Compare throughput, completed requests, TTFT, TPOT, ITL, and end-to-end latency using equivalent workloads.

## Post-process traces with TraceLens

Enable TraceLens in the profiled benchmark YAML; torch traces are its required input:

```yaml
benchmark:
  profiler:
    torch_profiler:
      enabled: true
    tracelens:
      enabled: true
      analysis_mode: inference
      analysis_stages: all
      export_format: csv
```

Use `analysis_mode: inference` for vLLM/SGLang. It splits the rank-0 trace into `prefilldecode`, `decode`, and `prefill` stages when available, runs TraceLens post-processing, and writes full stage reports plus compact `*_kernel_roofline_simple.csv` files under the benchmark workspace's `tracelens/` directory. For direct PyTorch trace reporting, use `analysis_mode: pytorch`.

Open the compact roofline CSVs first. Rank rows by `kernel_time_ms_sum` or `time_pct`; then use `roofline_bound`, arithmetic intensity, achieved TFLOP/s or TB/s, and `pct_roofline_mean` to form an optimization hypothesis. Confirm `benchmark_report.json.tracelens_analysis` has outputs and no error before treating post-processing as successful. Use `analysis_mode: pytorch` when the task specifically needs the legacy direct single-rank or multi-rank collective reports.

Magpie's integrated TraceLens stage produces CSV/Excel analysis artifacts, not an agent-written `analysis.md`. If the user requests a prioritized agentic report, pass the captured trace to the separate `tracelens-analysis-orchestrator` skill when installed; keep that result distinct from Magpie's benchmark report.

## Analyze existing traces and find source

Run standalone gap analysis with `--trace-dir` directly on `benchmark`:

From amd/skills, MIT.

More from amd/skills