ascend-profiling-anomaly

Files (6)

• SKILL.md
• architecture_report_template.md · references/
• kernel_data_guide.md · references/
• rulebook.md · references/
• schema.json · references/
• reference_host_gap_branch.py · scripts/

SKILL.md (first lines)

# Ascend Profiling Anomaly Discovery Skill

## Purpose

Analyze Ascend NPU profiling data through three parallel pipelines:

1. **Structure breakdown**: step → structure (layer) → block / side → op → PMU judgement — answers *where the time goes*.
2. **Anomaly discovery**: step → device busy union → bubble detection → anomaly tags → soft attribution — answers *what looks unnatural and where hidden issues may lurk*.
3. **Model architecture analysis**: FIA timeline → pass boundaries → layer classification → per-layer sub-structure → communication pipeline → architecture summary — answers *what is this model and how does each component execute*. **Produces a separate Markdown report file.**

The core philosophy is **separation of concerns**: "anomaly exists" is a hard fact derived from device intervals; "why it exists" is a soft attribution that may require additional evidence. Even under weak profiling configurations (no stacks, no shapes, sparse host events), the skill must still reliably surface device idle bubbles and risk labels.

## Reference Files — When to Read

Read these before starting analysis:

| File | When to read | What it contains |
|---|---|---|
| `references/kernel_data_guide.md` | **Always — read first** | Raw data column schemas for kernel_details.csv, op_summary, trace_view.json; the step → structure → block/side → op hierarchy; how to parse, filter, assign kernels at each level; multi-stream handling; per-level timing aggregation |
| `references/rulebook.md` | Always | Anomaly thresholds, tagging rules, decision tables, soft attribution rules, AICPU classification, wait-anchor rules |
| `references/architecture_report_template.md` | **Always — read before producing the architecture report** | Full template for the standalone Markdown architecture report: required sections, formatting rules, analysis techniques for layer classification, communication overlap measurement, per-layer timing breakdowns |
| `references/schema.json` | When producing structured JSON output | Full JSON schema for the `anomaly_discovery` output object |
| `scripts/reference_host_gap_branch.py` | When writing analysis scripts | Reference Python implementation for interval merging, bubble metrics, soft attribution, wait-anchor scoring |

## Pipeline Overview

The full state machine:

```
INGEST → INVENTORY → FACT_EXTRACTION → CANDIDATE_STEP_DETECTION →
STEP_GROUPING → MACRO_STEP_RESOLUTION → SEGMENTATION →
CLOCK_ACCOUNTING → ANOMALY_DISCOVERY → PERF_JUDGEMENT →
RECOMMENDATION → RENDER → DONE
                                        ↘
                          ARCHITECTURE_ANALYSIS → ARCH_REPORT_RENDER → ARCH_REPORT_SAVE
```

`ANOMALY_DISCOVERY` sits after `CLOCK_ACCOUNTING` and before `PERF_JUDGEMENT`. It receives already-segmented steps and structures, and runs the bubble detection pipeline on top of them.

`ARCHITECTURE_ANALYSIS` runs in parallel with `PERF_JUDGEMENT`, using the same segmented data from `CLOCK_ACCOUNTING` plus FIA timeline analysis. It produces a **separate Markdown report file** saved alongside the profiling data.

---

## Data Hierarchy: Step → Structure → Block/Side → Op

Understanding how raw kernel data maps to each level is essential. Read `references/kernel_data_guide.md` for the full column schemas and parsing details. Here is the conceptual overview:

### Level 0: Raw Kernels

Each row in `kernel_details.csv` represents a single device kernel execution — one invocation of an AI Core task, an AI CPU task, or an HCCL communication task. Key fields: `Name`, `Task Type`, `Start Time(us)`, `Duration(us)`, `Wait Time(us)`, `Accelerator Core`, `Stream ID`, `Input Shapes`, `Output Shapes`.

### Level 1: Step

A step is one training/inference iteration. Steps are identified by `ProfilerStep#N` user annotations or `Iteration#N` markers in `trace_view.json`. Each step defines a **service window** `[S_i, S_{i+1})`. All kernels whose start time falls within this window belong to step `i`.

At step level, compute:
- Total service time, device busy union, underfeed ratio
- Prelaunch gap, tail gap, internal bubbles
- Per-step anomaly tags and soft root-cause labels

### Level 2: Structure (Layer)

Within a step, kernels form repeating **structures** — typically corresponding to model layers (e.g., transformer blocks, attention layers, MLP blocks). Segmentation identifies these by:

1. Repeating name-pattern sequences in the kernel timeline
2. Significant time gaps between kernel groups
3. User annotations marking layer boundaries (if present)

Each structure contains a contiguous span of kernels within the step window.

At structure level, compute:
- Structure wall time, device busy union within structure span
- Structure share of total step time
- Per-structure bubble metrics (is the bubble inside this structure or between structures?)

### Level 3: Block / Side

Within each structure, kernels split into:

- **Block (main compute path)**: The dominant chain of AI Core kernels forming the forward or backward pass of this layer. These execute on the main compute stream.
- **Side (auxiliary ops)**: Everything else — small element-wise ops, communication (HCCL), AI CPU fallback ops, memory copies, synchronization events. These may execute on separate streams.

At block/side level, maintain **four timing perspectives** simultaneously:
- `wall_ms` — wall-clock time from first kernel start to last kernel end
- `busy_union_ms` — merged device-busy time (accounts for multi-stream overlap)
- `kernel_sum_ms` — arithmetic sum of all kernel durations (ignores overlap)
- `total_cost_ms` — sum of `duration + wait` for all kernels

Conclusions based on only one metric are incomplete. A block appearing heavy in `kernel_sum` but light in `wall` means high stream parallelism. A side appearing heavy in `total_cost` but light in `duration` means wait-anchor false hotspot risk.

### Level 4: Op (individual operator)

The finest grain. Each op may produce one or many device kernels. Op-level analysis handles:
- Top ops by total cost vs. top ops by kernel duration (these rankings often differ)
- Wait-anchor detection: ops with `wait_ratio > 0.95` and tiny `duration` but high `total_cost`
- AICPU classification: ops running on AI CPU instead of AI Core, classified by `masked_ratio`
- Small-op initial judgement: whether small individual ops are real inefficiencies or noise

---

## Anomaly Discovery Pipeline (Detail)

### Phase 1: BUILD_DEVICE_INTERVALS

For each step, collect all device kernel intervals from `kernel_details.csv`:

```
device_intervals = []
for each kernel row where Start_Time_us is within step window:
    s = max(step_start_us, row.Start_Time_us)
    e = min(step_end_us, row.Start_Time_us + row.Duration_us)
    if e > s:
        device_intervals.append(Interval(s, e))
```

Key rules:
- Clip kernel intervals to the step window boundary
- Apply communication dedup rules BEFORE interval statistics (see `references/kernel_data_guide.md` section on comm dedup)
- Include AI_CORE, AI_CPU, and HCCL tasks — all count as "device busy"

From Ascend/agent-skills, MulanPSL-2.0.

More from Ascend/agent-skills