gke-ai-troubleshooting-tpu-vm-repair-stuck
Diagnose GKE TPU VM instances stuck in Compute Engine repair (REPAIRING state) after host errors, maintenance events, or faulty host reports in single-host…
Skill by googleOfficial
Chalk Labs
Need help deploying gke-ai-troubleshooting-tpu-vm-repair-stuck?
Chalk Labs sets up agents like this inside real teams: connected to your tools, tested on your data, with a human in the loop where it matters.
Install
Works with Claude Code, Claude apps, Codex, Cursor, Gemini CLI, GitHub Copilot, VS Code, OpenCode, Goose, Windsurf. Read the repository before you give any tool access to your data.
npx agentmarket install skill/google-gke-ai-troubleshooting-tpu-vm-repair-stuck
Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.
Get the weekly diff
The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.
About
Use cases: security
Reviews
Write a reviewNo reviews yet. Installed it? Reviews come from verified installers only. How reviews work
Permissions
What this skill can do and what it asks for, from its SKILL.md. Checked Oct 25, 2026.
- Tools
- No tools found
- Secrets
- None asked for
- Install scripts
- None found
Network hosts it talks to 1
codeload.github.com· found in install command
Files it touches
~/.claude/skills/gke-ai-troubleshooting-tpu-vm-repair-stuck
What's inside
- Skill name
gke-ai-troubleshooting-tpu-vm-repair-stuck- Description
- Diagnose GKE TPU VM instances stuck in Compute Engine repair (REPAIRING state) after host errors, maintenance events, or faulty host reports in single-host, multi-host, dynamic slicing, and TPU All Capacity mode node pools. Use when TPU instances in a slice node pool remain in REPAIRING, host error events appear in audit logs, or TPU slices are unschedulable due to a VM in repair. Don't use for nodes reporting Ready: Unknown when no VM is in REPAIRING (use gke-ai-troubleshooting-node-unresponsive-timeout), other NotReady nodes when no VM is in REPAIRING (use gke-node-notready), Pending Pods when all nodes are Ready (use gke-workload-troubleshooting), JobSet preemption events (use gke-ai-troubleshooting-jobset-interruption), or Slice custom resource lifecycle failures (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring).
- License
- Apache-2.0
- Metadata
- version: 1.0.1 · category: Containers
- Instructions
- 365 lines in SKILL.md · view on GitHub
Files (2)
- SKILL.md
- attribution.md · references/
SKILL.md (first lines)
# Troubleshoot GKE TPU VMs stuck in Compute Engine repair (`REPAIRING`)
This skill checks the VM status, the host events, and the GKE repair status, and
routes recovery through GKE instead of direct Compute Engine changes.
## Attribution
**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-vm-repair-stuck)"`
and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-vm-repair-stuck)` to `User-Agent` headers.
## Prerequisites
- **Tools**:
- For installing and initializing `gcloud` (which covers authentication and
project selection), follow
[Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt).
- For installing `kubectl` and required plugins, follow
[Install kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)
and
[Install required plugins](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt).
- **Cloud Billing & Project Configuration**: Ensure the project has an active
Cloud Billing account. Follow
[Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)
to initialize `gcloud` and enable the Google Kubernetes Engine API
(`container.googleapis.com`), ensure the Compute Engine API
(`compute.googleapis.com`) and Cloud Logging API (`logging.googleapis.com`)
are enabled, and follow
[Store cluster information for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)
to configure `kubectl` cluster credentials.
- **Required IAM Roles**:
- Kubernetes Engine Viewer (`roles/container.viewer`)
- Compute Viewer (`roles/compute.viewer`)
- Logs Viewer (`roles/logging.viewer`)
- For the user who enables node auto-repair in Step 5, item 1: Kubernetes
Engine Cluster Admin (`roles/container.clusterAdmin`)
- **Documentation**:
- [View information about MIGs and managed instances](https://docs.cloud.google.com/compute/docs/instance-groups/getting-info-about-migs.md.txt)
(Sections: "List MIGs", "Check the status of managed instances")
- [Compute Engine instance lifecycle](https://docs.cloud.google.com/compute/docs/instances/instance-lifecycle.md.txt)
(Sections: "`REPAIRING` state")
- [Troubleshooting VM shutdowns and reboots](https://docs.cloud.google.com/compute/docs/troubleshooting/troubleshooting-reboots.md.txt)
(Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")
- [Configure cluster access for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)
(Sections: "Install kubectl", "Install required plugins", "Store cluster
information for kubectl")
- [Add and manage node pools](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-pools.md.txt)
(Sections: "View node pools in a Standard cluster", "Deploy a Pod to a
specific node pool")
- [Troubleshoot NodeNotReady state](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-notready.md.txt)
(Sections: "Check the node's status and conditions", "Confirm node
preemption")
- [About TPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/tpus.md.txt)
(Sections: "Multi-host TPU slice node pools")
- [Use dynamic slicing with a custom scheduler](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)
(Sections: "Requirements", "Verify the status of the nodes and the
partitions", "Monitor the status of the Slice custom resource", "Hardware
maintenance and failures")
- [About GKE dynamic slicing](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing.md.txt)
(Sections: "Key characteristics of dynamic slicing configurations")
- [Deploy TPUs in All Capacity mode in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpu-all-capacity-mode.md.txt)
(Sections: "Manage node failures", "Node failure and recovery")
- [Report and repair faulty TPU hosts in All Capacity mode](https://docs.cloud.google.com/tpu/docs/all-capacity-report-faulty-hosts.md.txt)
(Sections: "Understand the faulty host report process", "Review report
faulty host operations")
- [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)
(Sections: "Settings for Autopilot and Standard", "Repair criteria",
"Node repair process", "Verify node auto-repair is enabled for a Standard
node pool", "Enable auto-repair for an existing Standard node pool", "Get
information about recent automated repair events", "Node auto repair in TPU
slice nodes")
- [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)
(Sections: "Before you begin", "Configure auto repair for TPU slice nodes")
- [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)
(Sections: "Before you contact support", "Recommendations for efficient
support")
- [REST Resource: projects.locations.clusters.nodePools](https://docs.cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters.nodePools.md.txt)
(Sections: "Resource: NodePool")
> **Read-only rule**: Run read-only diagnostic commands only. Never drain,
> delete, or re-create nodes, or run any other command that changes the cluster.
> Give the user any fix to apply themselves.
When you recommend a fix, link the doc section that describes it.
---
## Diagnostic workflow
### Step 0: Collect context `[Low Risk]`
Collect the target parameters:
- `{project_id}`: Google Cloud project ID
- `{cluster_name}`: GKE cluster name
- `{location}`: Cluster region or zone
- `{cluster_mode}`: Cluster mode (`Autopilot` or `Standard`)
- `{tpu_version}`: TPU version (such as Ironwood `tpu7x`, TPU v6e, v5p, v5e, or
v4) and whether the node pool is a single-host or multi-host TPU slice node
pool
- `{nodepool_name}`: TPU node pool name
- `{group_name}` and `{zone}`: Name and zone of the managed instance group (MIG)
that backs the node pool:
- In a **Standard** cluster, follow
[View node pools in a Standard cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-pools.md.txt)
to describe `{nodepool_name}` and read its `instanceGroupUrls` field (see
[Resource: NodePool](https://docs.cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters.nodePools.md.txt)).
- In an **Autopilot** cluster (or if `{nodepool_name}` isn't known), follow
[List MIGs](https://docs.cloud.google.com/compute/docs/instance-groups/getting-info-about-migs.md.txt)
to list the managed instance groups in `{project_id}` and find
`{group_name}` and `{zone}`.
- `{instance_name}`: Affected TPU VM name, if known. If the user doesn't know
`{instance_name}`, Step 1 finds the instances in `REPAIRING`.
- `{issue_time}`: Incident timestamp in RFC3339 UTC, if known
---From google/skills, Apache-2.0.
Signals
- Stars
- 21,542
- Forks
- 1,821
- Open issues
- 21
- Last commit (this folder)
- Oct 19, 2026 (1 month ago)
From GitHub (fetched Oct 24, 2026).
Details
- Version
- 1.0.1
- License
- Apache-2.0
- Repository
- github.com/google/skills
- First seen
- Oct 24, 2026
- Last checked
- Oct 24, 2026
- Source
- GitHubgithub-graphql
Done for you
Rather not set it up yourself?
Chalk Labs scopes, builds and runs agents for your team: the task audit, the eval harness, the integrations and the weekly ops report. Fixed scope, priced before we start.
See how we build AI agents →Read the case studies
Chalk Labs is a two-founder AI and growth studio. We build AI agents, CRMs and MVPs for Web3 and AI companies, and have shipped 50+ product and token launches.
- launches shipped
- 50+
- launches shipped
- media placements
- 500+
- media placements
- founders on every account
- 2
- founders on every account
Press we've placed: Forbes · CoinDesk · Decrypt · Cointelegraph · The Block
Rahil Jain, Co-founder, Chalk Labs. Ex-Goldman Sachs engineer. You talk to the person who builds it.
Get gke-ai-troubleshooting-tpu-vm-repair-stuck running in your business
Tell us what you want to automate. A founder replies within one working day with a plan, not a pitch deck.
Cite this page
gke-ai-troubleshooting-tpu-vm-repair-stuck skill. Agent Mart. https://agentmart.dev/skills/google-gke-ai-troubleshooting-tpu-vm-repair-stuck (last updated 2026-10-24)