gke-ai-troubleshooting-node-unresponsive-timeout

Files (2)

• SKILL.md
• attribution.md · references/

SKILL.md (first lines)

# Troubleshoot unresponsive GKE TPU and GPU nodes (`NodeStatusUnknown`)

When the Compute Engine host of a TPU or GPU node has a fatal hardware error,
kernel panic, or non-maskable interrupt (NMI) lockup, the guest OS stops
responding. The kubelet can no longer send heartbeats, so the node `Ready`
condition becomes `Unknown` with `Reason: NodeStatusUnknown` (`Kubelet stopped
posting node status.`). If node auto-repair is disabled on the node pool, GKE
doesn't repair the node. The node can stay `NotReady`, and pods on it can stay
in `Terminating`, which blocks multi-host `JobSet` workloads from recovering.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-node-unresponsive-timeout)"`
and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-node-unresponsive-timeout)` to `User-Agent` headers.

## Prerequisites

- **Tools**: Install the
  [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) (`gcloud`) and
  `kubectl`.
- **Cloud Billing & Project Configuration**: Verify an active billing account is
  linked (`gcloud billing projects describe {project_id}`), authenticate
  (`gcloud auth login`), set the target project (`gcloud config set project
  {project_id}`), and ensure `container.googleapis.com`,
  `compute.googleapis.com`, `logging.googleapis.com`, and
  `monitoring.googleapis.com` are enabled.
- **Required IAM Roles**:
  - Kubernetes Engine Viewer (`roles/container.viewer`)
  - Compute Viewer (`roles/compute.viewer`)
  - Logs Viewer (`roles/logging.viewer`)
  - Monitoring Viewer (`roles/monitoring.viewer`)
  - For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin
    (`roles/container.clusterAdmin`)
- **Documentation**:
  - [Troubleshoot nodes with the NotReady status in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-notready.md.txt)
    (Sections: "Check the node's status and conditions", "Confirm node
    preemption", "Verify that the node has recovered")
  - [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)
    (Sections: "Monitor health metrics for TPU nodes and node pools", "Configure
    auto repair for TPU slice nodes")
  - [Troubleshoot OOM events](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/oom-events.md.txt)
  - [View GKE logs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs.md.txt)
    (Sections: "System logs")
  - [Viewing serial port output](https://docs.cloud.google.com/compute/docs/troubleshooting/viewing-serial-port-output.md.txt)
  - [Troubleshoot Linux VM boot issues due to kernel panic](https://docs.cloud.google.com/compute/docs/troubleshooting/kernel-panic.md.txt)
  - [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)
    (Sections: "Settings for Autopilot and Standard", "Verify node auto-repair
    is enabled for a Standard node pool", "Get information about recent
    automated repair events", "Enable auto-repair for an existing Standard node
    pool", "Repair criteria", "Node repair process", "Node auto repair in TPU
    slice nodes")
  - [Troubleshooting VM shutdowns and reboots](https://docs.cloud.google.com/compute/docs/troubleshooting/troubleshooting-reboots.md.txt)
    (Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")

> **Read-only rule**: Run read-only diagnostic commands only. Never drain,
> delete, or re-create nodes, or run any other command that changes the cluster.
> Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

---

## Diagnostic workflow

### Step 0: Collect context and set the investigation window `[Low Risk]`

Collect the target parameters. By default, query a 60-minute window `[T - 30m, T
+ 30m]` around `{issue_time}`:

- `{project_id}`: Google Cloud project ID
- `{cluster_name}`: GKE cluster name
- `{location}`: Cluster region or zone
- `{nodepool_name}`: Target TPU or GPU node pool name
- `{node_name}`: Unresponsive GKE node name (and its Compute Engine `{zone}`)
- `{issue_time}`: Incident timestamp in RFC3339 UTC
- `{start_time}`: `{issue_time} - 30m`
- `{end_time}`: `{issue_time} + 30m`

---

### Step 1: Verify the `NodeStatusUnknown` heartbeat timeout `[Low Risk]`

1. **Check Kubernetes node conditions**: To inspect the node status and verify
   whether the `Ready` condition is `Unknown` with `Reason: NodeStatusUnknown`
   (`Kubelet stopped posting node status.`), follow the instructions in the
   section
   [Check the node's status and conditions](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-notready.md.txt).
2. **Query Cloud Logging (read-only LQL)**: Query `k8s_node` and `k8s_cluster`
   logs across `[{start_time}, {end_time}]` to confirm when the control plane
   lost heartbeat contact with `{node_name}`:

```
(resource.type="k8s_node" OR resource.type="k8s_cluster")
resource.labels.cluster_name="{cluster_name}"
("{node_name}" AND ("NodeNotReady" OR "NodeStatusUnknown" OR "Kubelet stopped posting node status"))
timestamp >= "{start_time}" AND timestamp <= "{end_time}"
```

3. **Query Cloud Monitoring (read-only PromQL)**: Correlate the duration of the
   `Unknown` state using the GKE system metric
   `kubernetes.io/node/status_condition` (`kubernetes_io:node_status_condition`,
   GKE `1.32.1-gke.1357001+`) documented in
   [Monitor health metrics for TPU nodes and node pools](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt),
   filtered by `condition="Ready"` and `status="Unknown"`:

```promql
kubernetes_io:node_status_condition{
  monitored_resource="k8s_node",
  cluster_name="{cluster_name}",
  node_name="{node_name}",
  condition="Ready",
  status="Unknown"
}
```

From google/skills, Apache-2.0.

More from google/skills