gke-ai-troubleshooting-tpu-performance-degradation

Files (6)

• SKILL.md
• attribution.md · references/
• path-a-workload-bottlenecks.md · references/
• path-b-infrastructure-throttling.md · references/
• path-c-no-detected-analyzer.md · references/
• path-d-healthy-telemetry.md · references/

SKILL.md (first lines)

# Troubleshoot GKE TPU performance degradation with ML Diagnostics Workload Monitoring

Diagnose and mitigate Cloud TPU training throughput drops and step-time
regressions (`15%+` drop in TPU duty cycle) on Google Kubernetes Engine (GKE) by
correlating **ML Diagnostics Workload Monitoring** `MonitoredEvent` analyzer
reports with **1-minute Cloud Monitoring system metrics** and GKE node topology
labels.

---

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-performance-degradation)"`
and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-performance-degradation)` to `User-Agent` headers.

## Prerequisites

- **Tools**: Install the
  [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) (`gcloud` with
  `alpha` component for `gcloud alpha mldiagnostics`) and `kubectl`.
- **Cloud Billing & Project Configuration**: Verify an active billing account is
  linked (`gcloud billing projects describe {project_id}`), authenticate
  (`gcloud auth login`), set the target project (`gcloud config set project
  {project_id}`), and ensure `container.googleapis.com`,
  `monitoring.googleapis.com`, and `hypercomputecluster.googleapis.com` are
  enabled.
- **Supported workloads and GKE versions**: Google Cloud ML Diagnostics only
  supports JAX on TPUs (see
  [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)).
  Workload Monitoring is enabled by default, supports the `jobset` and `job` GKE
  job types, and is compatible with GKE versions `1.36.0-gke.4681000` and later,
  as stated in
  [Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt).
  If a workload uses another framework (such as PyTorch) or another custom
  resource type, `gcloud alpha mldiagnostics` won't list ML runs or monitored
  events for it. On-demand profiling (Step 4 Path A) additionally requires the
  cluster setup described in that document.
- **Required IAM Roles**:
  - Cluster Director Editor (`roles/hypercomputecluster.editor`), the role
    listed in the "IAM permissions" section of
    [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt),
    for the ML Diagnostics CLI and API calls in this skill (ML runs, monitored
    events, and on-demand profiler sessions)
  - Monitoring Viewer (`roles/monitoring.viewer`) for the PromQL queries in Step
    2
  - Kubernetes Engine Viewer (`roles/container.viewer`) for the `kubectl get
    nodes` query in Step 3
  - For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin
    (`roles/container.clusterAdmin`)
- **Reference Documentation**:
  - [ML Diagnostics platform](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/overview.md.txt)
    (Sections: "IAM permissions")
  - [Configure GKE for ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/gke.md.txt)
    (Sections: "Set up with gcloud CLI, Google Cloud console, or Terraform",
    "Manual installation", "Connection-operator")
  - [Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)
    (Sections: "Workload Monitoring and analyzers", "HBM Capacity Analyzer",
    "Host Memory Utilization Analyzer", "CPU Utilization Analyzer", "Access
    Workload Monitoring information through the API", "List all monitored
    events", "System Metrics")
  - [Get started with the ML Diagnostics CLI](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/cli.md.txt)
    (Sections: "List machine learning runs", "Monitored-events commands", "List
    profiler targets", "Capture on-demand profiler sessions")
  - [Scale container resource requests and limits](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/vertical-pod-autoscaling.md.txt)
    (Sections: "Identify workloads without resource requests or limits")
  - [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)
    (Sections: "Repair criteria", "Verify node auto-repair is enabled for a
    Standard node pool", "Enable auto-repair for an existing Standard node
    pool", "Node auto repair in TPU slice nodes")
  - [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)
    (Sections: "Configure auto repair for TPU slice nodes")
  - [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)
    (Sections: "Before you contact support")

> **Read-only rule**: Run read-only diagnostic commands only. Never drain,
> delete, or re-create nodes, or run any other command that changes the cluster.
> Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

---

## Analyzer routing overview

By default, Workload Monitoring treats a 15% drop in TPU duty cycle as a
performance degradation, raises a `MonitoredEvent` (`type:
PERFORMANCE_DEGRADATION`), and runs its analyzers. Consult the
[Workload Monitoring and analyzers](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)
section in the official Cloud TPU documentation for the complete analyzer
definitions, detection criteria, and subsections, and route remediation by
category:

- **Category A: workload resource bottlenecks (never cordon or replace nodes)**:
  - When the firing analyzer (`detectionState: "DETECTED"`) indicates a workload
    memory or host CPU capacity bottleneck (see the "HBM Capacity Analyzer",
    "Host Memory Utilization Analyzer", and "CPU Utilization Analyzer" sections
    of
    [Workload monitoring with ML Diagnostics](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)),
    route to
    [Path A: Workload resource bottlenecks](references/path-a-workload-bottlenecks.md)
    (workload optimization and profiling).
- **Category B: infrastructure and network fabric throttling (culprit nodes)**:
  - When the firing analyzer (`detectionState: "DETECTED"`) indicates an
    interconnect, thermal/power throttling, memory bandwidth, or Top-of-Rack
    (ToR) network fault on specific instances (see
    [Workload Monitoring and analyzers](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt)),
    route to
    [Path B: Infrastructure or network fabric throttling](references/path-b-infrastructure-throttling.md)
    (map the culprit Compute Engine instance IDs to GKE nodes, handle node
    repair through GKE, and escalate to Google Cloud Support).
- **Category C: `PERFORMANCE_DEGRADATION` event fired, but no analyzer reports
  `detectionState: "DETECTED"`**:
  - When a `PERFORMANCE_DEGRADATION` event exists (as in the sample output in
    [List all monitored events](https://docs.cloud.google.com/tpu/docs/ml-diagnostics/workload-monitoring.md.txt),
    where `analyzerReports` entries have `detectionState: "NOT_DETECTED"`),

From google/skills, Apache-2.0.

More from google/skills