Skip to content
Agentmart
Submit a tool

gke-ai-troubleshooting-tpu-vm-repair-stuck

Diagnose GKE TPU VM instances stuck in Compute Engine repair (REPAIRING state) after host errors, maintenance events, or faulty host reports in single-host…

Skill by googleOfficial

IndexedApache-2.0★ 21,542
Write a review

Chalk Labs

Need help deploying gke-ai-troubleshooting-tpu-vm-repair-stuck?

Chalk Labs sets up agents like this inside real teams: connected to your tools, tested on your data, with a human in the loop where it matters.

Book a call

Install

Works with Claude Code, Claude apps, Codex, Cursor, Gemini CLI, GitHub Copilot, VS Code, OpenCode, Goose, Windsurf. Read the repository before you give any tool access to your data.

npx agentmarket install skill/google-gke-ai-troubleshooting-tpu-vm-repair-stuck

Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

We use your email only for this newsletter. Privacy

About

Diagnose GKE TPU VM instances stuck in Compute Engine repair (REPAIRING state) after host errors, maintenance events, or faulty host reports in single-host, multi-host, dynamic slicing, and TPU All Capacity mode node pools. Use when TPU instances in a slice node pool remain in REPAIRING, host error events appear in audit logs, or TPU slices are unschedulable due to a VM in repair. Don't use for nodes reporting Ready: Unknown when no VM is in REPAIRING (use gke-ai-troubleshooting-node-unresponsive-timeout), other NotReady nodes when no VM is in REPAIRING (use gke-node-notready), Pending Pods when all nodes are Ready (use gke-workload-troubleshooting), JobSet preemption events (use gke-ai-troubleshooting-jobset-interruption), or Slice custom resource lifecycle failures (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring).

Use cases: security

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this skill can do and what it asks for, from its SKILL.md. Checked Oct 25, 2026.

Tools
No tools found
Secrets
None asked for
Install scripts
None found

Network hosts it talks to 1

  • codeload.github.com · found in install command

Files it touches

  • ~/.claude/skills/gke-ai-troubleshooting-tpu-vm-repair-stuck

What's inside

Skill name
gke-ai-troubleshooting-tpu-vm-repair-stuck
Description
Diagnose GKE TPU VM instances stuck in Compute Engine repair (REPAIRING state) after host errors, maintenance events, or faulty host reports in single-host, multi-host, dynamic slicing, and TPU All Capacity mode node pools. Use when TPU instances in a slice node pool remain in REPAIRING, host error events appear in audit logs, or TPU slices are unschedulable due to a VM in repair. Don't use for nodes reporting Ready: Unknown when no VM is in REPAIRING (use gke-ai-troubleshooting-node-unresponsive-timeout), other NotReady nodes when no VM is in REPAIRING (use gke-node-notready), Pending Pods when all nodes are Ready (use gke-workload-troubleshooting), JobSet preemption events (use gke-ai-troubleshooting-jobset-interruption), or Slice custom resource lifecycle failures (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring).
License
Apache-2.0
Metadata
version: 1.0.1 · category: Containers
Instructions
365 lines in SKILL.md · view on GitHub

Files (2)

  • SKILL.md
  • attribution.md · references/

SKILL.md (first lines)

# Troubleshoot GKE TPU VMs stuck in Compute Engine repair (`REPAIRING`)

This skill checks the VM status, the host events, and the GKE repair status, and
routes recovery through GKE instead of direct Compute Engine changes.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-vm-repair-stuck)"`
and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-vm-repair-stuck)` to `User-Agent` headers.

## Prerequisites

- **Tools**:
  - For installing and initializing `gcloud` (which covers authentication and
    project selection), follow
    [Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt).
  - For installing `kubectl` and required plugins, follow
    [Install kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)
    and
    [Install required plugins](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt).
- **Cloud Billing & Project Configuration**: Ensure the project has an active
  Cloud Billing account. Follow
  [Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)
  to initialize `gcloud` and enable the Google Kubernetes Engine API
  (`container.googleapis.com`), ensure the Compute Engine API
  (`compute.googleapis.com`) and Cloud Logging API (`logging.googleapis.com`)
  are enabled, and follow
  [Store cluster information for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)
  to configure `kubectl` cluster credentials.
- **Required IAM Roles**:
  - Kubernetes Engine Viewer (`roles/container.viewer`)
  - Compute Viewer (`roles/compute.viewer`)
  - Logs Viewer (`roles/logging.viewer`)
  - For the user who enables node auto-repair in Step 5, item 1: Kubernetes
    Engine Cluster Admin (`roles/container.clusterAdmin`)
- **Documentation**:
  - [View information about MIGs and managed instances](https://docs.cloud.google.com/compute/docs/instance-groups/getting-info-about-migs.md.txt)
    (Sections: "List MIGs", "Check the status of managed instances")
  - [Compute Engine instance lifecycle](https://docs.cloud.google.com/compute/docs/instances/instance-lifecycle.md.txt)
    (Sections: "`REPAIRING` state")
  - [Troubleshooting VM shutdowns and reboots](https://docs.cloud.google.com/compute/docs/troubleshooting/troubleshooting-reboots.md.txt)
    (Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")
  - [Configure cluster access for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)
    (Sections: "Install kubectl", "Install required plugins", "Store cluster
    information for kubectl")
  - [Add and manage node pools](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-pools.md.txt)
    (Sections: "View node pools in a Standard cluster", "Deploy a Pod to a
    specific node pool")
  - [Troubleshoot NodeNotReady state](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-notready.md.txt)
    (Sections: "Check the node's status and conditions", "Confirm node
    preemption")
  - [About TPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/tpus.md.txt)
    (Sections: "Multi-host TPU slice node pools")
  - [Use dynamic slicing with a custom scheduler](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)
    (Sections: "Requirements", "Verify the status of the nodes and the
    partitions", "Monitor the status of the Slice custom resource", "Hardware
    maintenance and failures")
  - [About GKE dynamic slicing](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing.md.txt)
    (Sections: "Key characteristics of dynamic slicing configurations")
  - [Deploy TPUs in All Capacity mode in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpu-all-capacity-mode.md.txt)
    (Sections: "Manage node failures", "Node failure and recovery")
  - [Report and repair faulty TPU hosts in All Capacity mode](https://docs.cloud.google.com/tpu/docs/all-capacity-report-faulty-hosts.md.txt)
    (Sections: "Understand the faulty host report process", "Review report
    faulty host operations")
  - [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)
    (Sections: "Settings for Autopilot and Standard", "Repair criteria",
    "Node repair process", "Verify node auto-repair is enabled for a Standard
    node pool", "Enable auto-repair for an existing Standard node pool", "Get
    information about recent automated repair events", "Node auto repair in TPU
    slice nodes")
  - [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)
    (Sections: "Before you begin", "Configure auto repair for TPU slice nodes")
  - [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)
    (Sections: "Before you contact support", "Recommendations for efficient
    support")
  - [REST Resource: projects.locations.clusters.nodePools](https://docs.cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters.nodePools.md.txt)
    (Sections: "Resource: NodePool")

> **Read-only rule**: Run read-only diagnostic commands only. Never drain,
> delete, or re-create nodes, or run any other command that changes the cluster.
> Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

---

## Diagnostic workflow

### Step 0: Collect context `[Low Risk]`

Collect the target parameters:

- `{project_id}`: Google Cloud project ID
- `{cluster_name}`: GKE cluster name
- `{location}`: Cluster region or zone
- `{cluster_mode}`: Cluster mode (`Autopilot` or `Standard`)
- `{tpu_version}`: TPU version (such as Ironwood `tpu7x`, TPU v6e, v5p, v5e, or
  v4) and whether the node pool is a single-host or multi-host TPU slice node
  pool
- `{nodepool_name}`: TPU node pool name
- `{group_name}` and `{zone}`: Name and zone of the managed instance group (MIG)
  that backs the node pool:
  - In a **Standard** cluster, follow
    [View node pools in a Standard cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-pools.md.txt)
    to describe `{nodepool_name}` and read its `instanceGroupUrls` field (see
    [Resource: NodePool](https://docs.cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters.nodePools.md.txt)).
  - In an **Autopilot** cluster (or if `{nodepool_name}` isn't known), follow
    [List MIGs](https://docs.cloud.google.com/compute/docs/instance-groups/getting-info-about-migs.md.txt)
    to list the managed instance groups in `{project_id}` and find
    `{group_name}` and `{zone}`.
- `{instance_name}`: Affected TPU VM name, if known. If the user doesn't know
  `{instance_name}`, Step 1 finds the instances in `REPAIRING`.
- `{issue_time}`: Incident timestamp in RFC3339 UTC, if known

---

From google/skills, Apache-2.0.

Signals

Stars
21,542
Forks
1,821
Open issues
21
Last commit (this folder)
Oct 19, 2026 (1 month ago)

From GitHub (fetched Oct 24, 2026).

Details

Version
1.0.1
License
Apache-2.0
First seen
Oct 24, 2026
Last checked
Oct 24, 2026

Done for you

Rather not set it up yourself?

Chalk Labs scopes, builds and runs agents for your team: the task audit, the eval harness, the integrations and the weekly ops report. Fixed scope, priced before we start.

See how we build AI agents →Read the case studies

Chalk Labs is a two-founder AI and growth studio. We build AI agents, CRMs and MVPs for Web3 and AI companies, and have shipped 50+ product and token launches.

launches shipped
50+
launches shipped
media placements
500+
media placements
founders on every account
2
founders on every account

Press we've placed: Forbes · CoinDesk · Decrypt · Cointelegraph · The Block

Rahil Jain, Co-founder, Chalk Labs. Ex-Goldman Sachs engineer. You talk to the person who builds it.

Get gke-ai-troubleshooting-tpu-vm-repair-stuck running in your business

Tell us what you want to automate. A founder replies within one working day with a plan, not a pitch deck.

Goes to the Chalk Labs founders, not the newsletter. Privacy

Need help deploying gke-ai-troubleshooting-tpu-vm-repair-stuck?

CallGet help
Cite this page

gke-ai-troubleshooting-tpu-vm-repair-stuck skill. Agent Mart. https://agentmart.dev/skills/google-gke-ai-troubleshooting-tpu-vm-repair-stuck (last updated 2026-10-24)