gke-node-notready

Files (2)

• SKILL.md
• attribution.md · references/

SKILL.md (first lines)

# GKE Node NotReady Troubleshooting Skill

Use this skill to systematically diagnose why one or more GKE nodes report a
`NotReady` (or `Ready: Unknown`) status and to propose safe remediations. A
`NotReady` status means the node's kubelet is not reporting to the control plane
correctly, so Kubernetes stops scheduling new Pods on the node, which can reduce
application capacity and cause downtime.

This skill operates **non-interactively** and enforces a **read-only diagnostics
boundary**: gather evidence first, then propose a fix (a `kubectl`/`gcloud`
command or a GitOps manifest change) for a human to apply. **Never** mutate the
cluster, drain, delete, or recreate nodes automatically.

> [!IMPORTANT]
> First rule out an **expected** `NotReady`: a node that is newly provisioning,
> upgrading, being repaired, cordoned, or scaling down will transiently report
> `NotReady`. Only treat it as a fault if it persists beyond the expected window.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-node-notready)"`
and append `agent-skills/1.0 (skill:gke-node-notready)` to `User-Agent` headers.

## 🔍 Diagnostic Workflow

### Step 0: Context discovery & time window

1.  **Parameter extraction** — obtain `project_id`, `cluster_name`,
    `cluster_location`, and `node_name` non-interactively from the user prompt,
    active `SETTINGS.md`, or environment defaults (`kubectl config current-context`,
    `gcloud config get-value project`).
2.  **Credentials & fallback** — attempt
    `gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id}`.
    If the cluster is unreachable or commands fail (sandbox/dry-run/offline),
    present the exact diagnostic commands for a human to run and continue the
    analysis from the reported symptoms.
3.  **Time window** — determine `{issue_time}` (explicit, relative, or now) and
    center a 1-hour window around it (`start = issue_time - 30m`,
    `end = issue_time + 30m`) for all log/metric queries.

--------------------------------------------------------------------------------

### Step 1: Identify NotReady nodes and gather initial status

```bash
# List nodes and spot NotReady status, node IPs, and container-runtime version.
kubectl get nodes -o wide

# Inspect the affected node's Conditions and Events (the primary clues).
kubectl describe node "{node_name}"
```

Equivalent via Cloud Logging (preferred when kubectl access is limited or for
historical events). Open it as a **Logs Explorer deep link** — URL-encode the
query and append the project and Step 0 time window:
`https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}`
(encode `/` as `%2F`, or use `;duration=PT1H` for a rolling hour):

```
resource.type="k8s_node"
log_id("events")
resource.labels.node_name="{node_name}"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
```

**Interpret the `Conditions` table:**

- `Ready: False` / `Ready: Unknown` with reason `KubeletNotReady` /
  `NodeStatusUnknown` ("Kubelet stopped posting node status") → kubelet or
  runtime problem; continue to Step 2.
- `MemoryPressure: True`, `DiskPressure: True`, `PIDPressure: True` → resource
  exhaustion; go to Step 4b.
- `NetworkUnavailable: True` → networking/CNI problem; go to Step 4d.

--------------------------------------------------------------------------------

### Step 2: Scan kubelet logs for error signatures

Open these kubelet logs as a **Logs Explorer deep link** using the same
`logs/query;query={URL_ENCODED_QUERY};timeRange=...?project=...` pattern as Step 1.

```
resource.type="k8s_node"
resource.labels.node_name="{node_name}"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
log_id("kubelet")
severity>=WARNING
```

Also review the node's serial-console logs (`log_id("serialconsole.googleapis.com/serial_port_1_output")`
or the `resource.type="gce_instance"` serial logs) for kernel `TaskHung`,
OOM-killer, or disk I/O errors that correlate with the kubelet failures.

--------------------------------------------------------------------------------

### Step 3: Map the signature to a root cause (decision table)

| Kubelet / event signature | Likely root cause | Go to |
| --- | --- | --- |
| `runtime is down`, `Container runtime not ready`, errors on `/run/containerd/containerd.sock` (connection refused / DeadlineExceeded) | Container runtime (`containerd`) down or unresponsive | Step 4a |
| `Got sys oom event from cadvisor` / kernel OOM-killer in serial logs | System (node-level) OOM killed critical processes | Step 4b |
| `PLEG is not healthy` | PLEG stalled, usually node overload (CPU/disk) | Step 4c |
| `TaskHung` for `containerd`/`kubelet`, high disk latency | Disk throttling / I/O starvation | Step 4b |
| `failed to ensure lease`, `leases.coordination.k8s.io ... namespace kube-node-lease ... terminating` | `kube-node-lease` termination → NotReady flapping | Step 4f |
| Kubelet cannot reach API server, TLS/dial timeouts | Kubelet ↔ control-plane connectivity | Step 4d |
| `NetworkPluginNotReady`, `cni plugin not initialized`, `NetworkUnavailable` | CNI plugin failure | Step 4d |
| Node-critical DaemonSet Pods (CNI, kube-proxy, metadata) blocked from admission | Admission webhook interference | Step 4e |
| Only generic `NodeNotReady`, no other signature | Cause unclear — widen to Step 4d, then escalate | Escalation |

--------------------------------------------------------------------------------

### Step 4: Branch investigations

From google/skills, Apache-2.0.

More from google/skills