gke-ai-troubleshooting-tpu-mxla-hang
Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA)…
Skill by googleOfficial
Chalk Labs
Need help deploying gke-ai-troubleshooting-tpu-mxla-hang?
Chalk Labs sets up agents like this inside real teams: connected to your tools, tested on your data, with a human in the loop where it matters.
Install
Works with Claude Code, Claude apps, Codex, Cursor, Gemini CLI, GitHub Copilot, VS Code, OpenCode, Goose, Windsurf. Read the repository before you give any tool access to your data.
npx agentmarket install skill/google-gke-ai-troubleshooting-tpu-mxla-hang
Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.
Get the weekly diff
The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.
About
Use cases: devops, data, ml
Reviews
Write a reviewNo reviews yet. Installed it? Reviews come from verified installers only. How reviews work
Permissions
What this skill can do and what it asks for, from its SKILL.md. Checked Oct 25, 2026.
- Tools
- No tools found
- Secrets
- None asked for
- Install scripts
- None found
Network hosts it talks to 1
codeload.github.com· found in install command
Files it touches
~/.claude/skills/gke-ai-troubleshooting-tpu-mxla-hang
What's inside
- Skill name
gke-ai-troubleshooting-tpu-mxla-hang- Description
- Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA) Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute Cloud Monitoring multi-slice latency metrics (`kubernetes.io/container/multislice/*`). Distinguishes XLA compiler/HLO launch divergence and host data-input stalls from TPU chip, SparseCore, ICI, or network fabric faults. Use when multi-slice TPU training jobs freeze without progressing steps, emit `HANG_DETECTED`, or stall in collective operations. Don't use for gradual step-time throughput drops without hangs (use gke-ai-troubleshooting-tpu-performance-degradation) or pod preemption/eviction restarts (use gke-ai-troubleshooting-jobset-interruption).
- License
- Apache-2.0
- Metadata
- version: 1.0.1 · category: AiAndMachineLearning
- Instructions
- 286 lines in SKILL.md · view on GitHub
Files (7)
- SKILL.md
- attribution.md · references/
- path-a-compiler-hlo-divergence.md · references/
- path-b-program-queueing-input-stall.md · references/
- path-c-hardware-network-faults.md · references/
- path-d-no-detected-or-unknown.md · references/
- path-e-healthy-telemetry.md · references/
SKILL.md (first lines)
# Troubleshoot GKE TPU multi-slice hangs with the MXLA Hang Analyzer
Diagnose Cloud TPU multi-slice training hangs on Google Kubernetes Engine (GKE)
by correlating Megascale `HANG_DETECTED` logs and **ML Diagnostics Workload
Monitoring** `Megascale XLA (MXLA) Hang Analyzer` reports with **1-minute
multi-slice latency metrics** (`kubernetes.io/container/multislice/*`) and GKE
node topology labels.
---
## Attribution
**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)"`
and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)` to `User-Agent` headers.
## Prerequisites
- **Tools**: Install the
[Google Cloud SDK](https://cloud.google.com/sdk/docs/install) (`gcloud` with
`alpha` component for `gcloud alpha mldiagnostics`) and `kubectl`.
- **Cloud Billing & Project Configuration**: Verify an active billing account is
linked (`gcloud billing projects describe {project_id}`), authenticate
(`gcloud auth login`), set the target project (`gcloud config set project
{project_id}`), and ensure `container.googleapis.com`,
`logging.googleapis.com`, `monitoring.googleapis.com`, and
`hypercomputecluster.googleapis.com` are enabled.
- **Supported workloads and versions**: Google Cloud ML Diagnostics only
supports JAX on TPUs (see
[ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)).
Workload Monitoring is enabled by default, supports the `jobset` and `job` GKE
job types, and is compatible with GKE versions `1.36.0-gke.4681000` and later
([Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt),
which also covers the cluster setup needed for on-demand profiling in Step 4
Path B). If a workload uses another framework (such as PyTorch) or another
custom resource type, `gcloud alpha mldiagnostics` won't list ML runs or
monitored events for it. The Megascale XLA hang analyzer and Megascale XLA
metrics require LibTPU `0.40.0` or later (see "Get started" in
[Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)).
- **Required IAM Roles**:
- Cluster Director Editor (`roles/hypercomputecluster.editor`), the role
listed in the "IAM permissions" section of
[ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt),
for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored
events, and on-demand profiler sessions)
- Monitoring Viewer (`roles/monitoring.viewer`) for the PromQL queries in Step
2
- Logs Viewer (`roles/logging.viewer`) for the Cloud Logging query in Step 1
- Kubernetes Engine Viewer (`roles/container.viewer`) for the `kubectl get
nodes` query in Step 3
- For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin
(`roles/container.clusterAdmin`)
- **Reference Documentation**:
- [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)
(Sections: "IAM permissions")
- [Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt)
(Sections: "Set up with gcloud CLI, Google Cloud console, or Terraform",
"Manual installation", "Connection-operator")
- [Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)
(Sections: "Get started", "Megascale XLA (MXLA) Hang Analyzer", "Access
Workload Monitoring information through the API", "System Metrics")
- [Get started with the ML Diagnostics CLI](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)
(Sections: "List machine learning runs", "Monitored-events commands", "List
profiler targets", "Capture on-demand profiler sessions")
- [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)
(Sections: "Before you contact support")
- [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)
(Sections: "Repair criteria", "Verify node auto-repair is enabled for a
Standard node pool", "Enable auto-repair for an existing Standard node
pool", "Node auto repair in TPU slice nodes")
- [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)
(Sections: "Configure auto repair for TPU slice nodes")
- [Dump HLO Computations (OpenXLA)](https://openxla.org/xla/hlo_dumps)
> **Read-only rule**: Run read-only diagnostic commands only. Never drain,
> delete, or re-create nodes, or run any other command that changes the cluster.
> Give the user any fix to apply themselves.
When you recommend a fix, link the doc section that describes it.
---
## MXLA Hang Analyzer routing overview
A Megascale hang occurs when a multi-slice worker has waited on a Megascale
communication operation for a set timeout period. The TPU logs then show a
Megascale `HANG_DETECTED` message. `HANG_DETECTED` is a catch-all signal that
the workload isn't progressing, and the cause can be in software or in hardware.
When a hang occurs, ML Diagnostics runs the `Megascale XLA (MXLA) Hang
Analyzer`, which reports the likely cause as a code. Consult the
[Megascale XLA (MXLA) Hang Analyzer](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)
section for the definition and recommended action of each code, and route by
category:
- **Category A: compiler or HLO divergence (never cordon or replace nodes)**:
- When the analyzer reports different HLO modules, inconsistent HLO
compilation, or an inconsistent launch order across VMs (such as
`FINGERPRINT_MISMATCH`), route to
[Path A: Compiler or HLO divergence](references/path-a-compiler-hlo-divergence.md).
- **Category B: host program queueing or data input stall (never cordon or
replace nodes)**:
- When the analyzer reports that VMs aren't queuing programs to the TPU or
that data input stalled (such as `DATA_INPUT_STALL`), route to
[Path B: Host program queueing or data input stall](references/path-b-program-queueing-input-stall.md).
- **Category C: hardware or network faults on specific instances**:
- When the analyzer attributes the hang to a TPU chip, SparseCore, ICI, or DCN
networking issue, or to an unrecoverable error on specific instances, route
to
[Path C: Hardware or network faults](references/path-c-hardware-network-faults.md).
- **Category D: hang signal or event exists, but the analyzer is `NOT_DETECTED`
or reports `UNKNOWN`**:
- When `HANG_DETECTED` logs or a hang monitored event fired, but the analyzer
report has `detectionState: "NOT_DETECTED"` or reports `UNKNOWN` ("The MXLA
hang was detected but a potential cause is not determined"), route to
[Path D: No DETECTED analyzer or UNKNOWN cause](references/path-d-no-detected-or-unknown.md).From google/skills, Apache-2.0.
Signals
- Stars
- 21,542
- Forks
- 1,821
- Open issues
- 21
- Last commit (this folder)
- Oct 19, 2026 (29 days ago)
From GitHub (fetched Oct 24, 2026).
Details
- Version
- 1.0.1
- License
- Apache-2.0
- Repository
- github.com/google/skills
- First seen
- Oct 24, 2026
- Last checked
- Oct 24, 2026
- Source
- GitHubgithub-graphql
Done for you
Rather not set it up yourself?
Chalk Labs scopes, builds and runs agents for your team: the task audit, the eval harness, the integrations and the weekly ops report. Fixed scope, priced before we start.
See how we build AI agents →Read the case studies
Chalk Labs is a two-founder AI and growth studio. We build AI agents, CRMs and MVPs for Web3 and AI companies, and have shipped 50+ product and token launches.
- launches shipped
- 50+
- launches shipped
- media placements
- 500+
- media placements
- founders on every account
- 2
- founders on every account
Press we've placed: Forbes · CoinDesk · Decrypt · Cointelegraph · The Block
Rahil Jain, Co-founder, Chalk Labs. Ex-Goldman Sachs engineer. You talk to the person who builds it.
Get gke-ai-troubleshooting-tpu-mxla-hang running in your business
Tell us what you want to automate. A founder replies within one working day with a plan, not a pitch deck.
Cite this page
gke-ai-troubleshooting-tpu-mxla-hang skill. Agent Mart. https://agentmart.dev/skills/google-gke-ai-troubleshooting-tpu-mxla-hang (last updated 2026-10-24)