Skip to content
Agentmart
Submit a tool

gke-ai-troubleshooting-tpu-mxla-hang

Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA)…

Skill by googleOfficial

IndexedApache-2.0★ 21,542
Write a review

Chalk Labs

Need help deploying gke-ai-troubleshooting-tpu-mxla-hang?

Chalk Labs sets up agents like this inside real teams: connected to your tools, tested on your data, with a human in the loop where it matters.

Book a call

Install

Works with Claude Code, Claude apps, Codex, Cursor, Gemini CLI, GitHub Copilot, VS Code, OpenCode, Goose, Windsurf. Read the repository before you give any tool access to your data.

npx agentmarket install skill/google-gke-ai-troubleshooting-tpu-mxla-hang

Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

We use your email only for this newsletter. Privacy

About

Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA) Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute Cloud Monitoring multi-slice latency metrics (`kubernetes.io/container/multislice/*`). Distinguishes XLA compiler/HLO launch divergence and host data-input stalls from TPU chip, SparseCore, ICI, or network fabric faults. Use when multi-slice TPU training jobs freeze without progressing steps, emit `HANG_DETECTED`, or stall in collective operations. Don't use for gradual step-time throughput drops without hangs (use gke-ai-troubleshooting-tpu-performance-degradation) or pod preemption/eviction restarts (use gke-ai-troubleshooting-jobset-interruption).

Use cases: devops, data, ml

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this skill can do and what it asks for, from its SKILL.md. Checked Oct 25, 2026.

Tools
No tools found
Secrets
None asked for
Install scripts
None found

Network hosts it talks to 1

  • codeload.github.com · found in install command

Files it touches

  • ~/.claude/skills/gke-ai-troubleshooting-tpu-mxla-hang

What's inside

Skill name
gke-ai-troubleshooting-tpu-mxla-hang
Description
Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA) Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute Cloud Monitoring multi-slice latency metrics (`kubernetes.io/container/multislice/*`). Distinguishes XLA compiler/HLO launch divergence and host data-input stalls from TPU chip, SparseCore, ICI, or network fabric faults. Use when multi-slice TPU training jobs freeze without progressing steps, emit `HANG_DETECTED`, or stall in collective operations. Don't use for gradual step-time throughput drops without hangs (use gke-ai-troubleshooting-tpu-performance-degradation) or pod preemption/eviction restarts (use gke-ai-troubleshooting-jobset-interruption).
License
Apache-2.0
Metadata
version: 1.0.1 · category: AiAndMachineLearning
Instructions
286 lines in SKILL.md · view on GitHub

Files (7)

  • SKILL.md
  • attribution.md · references/
  • path-a-compiler-hlo-divergence.md · references/
  • path-b-program-queueing-input-stall.md · references/
  • path-c-hardware-network-faults.md · references/
  • path-d-no-detected-or-unknown.md · references/
  • path-e-healthy-telemetry.md · references/

SKILL.md (first lines)

# Troubleshoot GKE TPU multi-slice hangs with the MXLA Hang Analyzer

Diagnose Cloud TPU multi-slice training hangs on Google Kubernetes Engine (GKE)
by correlating Megascale `HANG_DETECTED` logs and **ML Diagnostics Workload
Monitoring** `Megascale XLA (MXLA) Hang Analyzer` reports with **1-minute
multi-slice latency metrics** (`kubernetes.io/container/multislice/*`) and GKE
node topology labels.

---

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)"`
and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)` to `User-Agent` headers.

## Prerequisites

- **Tools**: Install the
  [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) (`gcloud` with
  `alpha` component for `gcloud alpha mldiagnostics`) and `kubectl`.
- **Cloud Billing & Project Configuration**: Verify an active billing account is
  linked (`gcloud billing projects describe {project_id}`), authenticate
  (`gcloud auth login`), set the target project (`gcloud config set project
  {project_id}`), and ensure `container.googleapis.com`,
  `logging.googleapis.com`, `monitoring.googleapis.com`, and
  `hypercomputecluster.googleapis.com` are enabled.
- **Supported workloads and versions**: Google Cloud ML Diagnostics only
  supports JAX on TPUs (see
  [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)).
  Workload Monitoring is enabled by default, supports the `jobset` and `job` GKE
  job types, and is compatible with GKE versions `1.36.0-gke.4681000` and later
  ([Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt),
  which also covers the cluster setup needed for on-demand profiling in Step 4
  Path B). If a workload uses another framework (such as PyTorch) or another
  custom resource type, `gcloud alpha mldiagnostics` won't list ML runs or
  monitored events for it. The Megascale XLA hang analyzer and Megascale XLA
  metrics require LibTPU `0.40.0` or later (see "Get started" in
  [Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)).
- **Required IAM Roles**:
  - Cluster Director Editor (`roles/hypercomputecluster.editor`), the role
    listed in the "IAM permissions" section of
    [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt),
    for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored
    events, and on-demand profiler sessions)
  - Monitoring Viewer (`roles/monitoring.viewer`) for the PromQL queries in Step
    2
  - Logs Viewer (`roles/logging.viewer`) for the Cloud Logging query in Step 1
  - Kubernetes Engine Viewer (`roles/container.viewer`) for the `kubectl get
    nodes` query in Step 3
  - For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin
    (`roles/container.clusterAdmin`)
- **Reference Documentation**:
  - [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)
    (Sections: "IAM permissions")
  - [Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt)
    (Sections: "Set up with gcloud CLI, Google Cloud console, or Terraform",
    "Manual installation", "Connection-operator")
  - [Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)
    (Sections: "Get started", "Megascale XLA (MXLA) Hang Analyzer", "Access
    Workload Monitoring information through the API", "System Metrics")
  - [Get started with the ML Diagnostics CLI](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)
    (Sections: "List machine learning runs", "Monitored-events commands", "List
    profiler targets", "Capture on-demand profiler sessions")
  - [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)
    (Sections: "Before you contact support")
  - [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)
    (Sections: "Repair criteria", "Verify node auto-repair is enabled for a
    Standard node pool", "Enable auto-repair for an existing Standard node
    pool", "Node auto repair in TPU slice nodes")
  - [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)
    (Sections: "Configure auto repair for TPU slice nodes")
  - [Dump HLO Computations (OpenXLA)](https://openxla.org/xla/hlo_dumps)

> **Read-only rule**: Run read-only diagnostic commands only. Never drain,
> delete, or re-create nodes, or run any other command that changes the cluster.
> Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

---

## MXLA Hang Analyzer routing overview

A Megascale hang occurs when a multi-slice worker has waited on a Megascale
communication operation for a set timeout period. The TPU logs then show a
Megascale `HANG_DETECTED` message. `HANG_DETECTED` is a catch-all signal that
the workload isn't progressing, and the cause can be in software or in hardware.
When a hang occurs, ML Diagnostics runs the `Megascale XLA (MXLA) Hang
Analyzer`, which reports the likely cause as a code. Consult the
[Megascale XLA (MXLA) Hang Analyzer](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)
section for the definition and recommended action of each code, and route by
category:

- **Category A: compiler or HLO divergence (never cordon or replace nodes)**:
  - When the analyzer reports different HLO modules, inconsistent HLO
    compilation, or an inconsistent launch order across VMs (such as
    `FINGERPRINT_MISMATCH`), route to
    [Path A: Compiler or HLO divergence](references/path-a-compiler-hlo-divergence.md).
- **Category B: host program queueing or data input stall (never cordon or
  replace nodes)**:
  - When the analyzer reports that VMs aren't queuing programs to the TPU or
    that data input stalled (such as `DATA_INPUT_STALL`), route to
    [Path B: Host program queueing or data input stall](references/path-b-program-queueing-input-stall.md).
- **Category C: hardware or network faults on specific instances**:
  - When the analyzer attributes the hang to a TPU chip, SparseCore, ICI, or DCN
    networking issue, or to an unrecoverable error on specific instances, route
    to
    [Path C: Hardware or network faults](references/path-c-hardware-network-faults.md).
- **Category D: hang signal or event exists, but the analyzer is `NOT_DETECTED`
  or reports `UNKNOWN`**:
  - When `HANG_DETECTED` logs or a hang monitored event fired, but the analyzer
    report has `detectionState: "NOT_DETECTED"` or reports `UNKNOWN` ("The MXLA
    hang was detected but a potential cause is not determined"), route to
    [Path D: No DETECTED analyzer or UNKNOWN cause](references/path-d-no-detected-or-unknown.md).

From google/skills, Apache-2.0.

Signals

Stars
21,542
Forks
1,821
Open issues
21
Last commit (this folder)
Oct 19, 2026 (29 days ago)

From GitHub (fetched Oct 24, 2026).

Details

Version
1.0.1
License
Apache-2.0
First seen
Oct 24, 2026
Last checked
Oct 24, 2026

Done for you

Rather not set it up yourself?

Chalk Labs scopes, builds and runs agents for your team: the task audit, the eval harness, the integrations and the weekly ops report. Fixed scope, priced before we start.

See how we build AI agents →Read the case studies

Chalk Labs is a two-founder AI and growth studio. We build AI agents, CRMs and MVPs for Web3 and AI companies, and have shipped 50+ product and token launches.

launches shipped
50+
launches shipped
media placements
500+
media placements
founders on every account
2
founders on every account

Press we've placed: Forbes · CoinDesk · Decrypt · Cointelegraph · The Block

Rahil Jain, Co-founder, Chalk Labs. Ex-Goldman Sachs engineer. You talk to the person who builds it.

Get gke-ai-troubleshooting-tpu-mxla-hang running in your business

Tell us what you want to automate. A founder replies within one working day with a plan, not a pitch deck.

Goes to the Chalk Labs founders, not the newsletter. Privacy

Need help deploying gke-ai-troubleshooting-tpu-mxla-hang?

CallGet help
Cite this page

gke-ai-troubleshooting-tpu-mxla-hang skill. Agent Mart. https://agentmart.dev/skills/google-gke-ai-troubleshooting-tpu-mxla-hang (last updated 2026-10-24)