pod-pending-debug

Files (1)

• SKILL.md

SKILL.md (first lines)

# Pod Scheduling Failure Diagnosis

When a pod is stuck in `Pending` state, follow this flow to identify why the scheduler cannot place it on a node.

**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify node taints, labels, or pod specs — that should be left to the user.

## Diagnostic Flow

### 1. Read the pod and its events — one call

```
k8s_inspect(kind: "pod", name: "<pod>", namespace: "<ns>")
```

The scheduler's `FailedScheduling` event carries the reason, and this returns the pod's events
alongside its phase and container states in a single call.

Two things to read carefully:
- The `FailedScheduling` message — it says how many nodes were evaluated and why each was rejected,
  and that count is what separates "no capacity anywhere" from "one taint on one node". The compact
  bundle keeps substantially more of long scheduler messages, but if a line ends in `…`, fetch that
  event with `kubectl get events` or `kubectl describe pod` before concluding from the missing tail.
- The final `status:` line. `partial (events: …)` means the events could not be read at all — nothing
  below applies until that is fixed.

An **empty** events section is not a conclusion. Events expire on a TTL (commonly one hour), so zero
events means *none are retained*, which covers two opposite situations:

- the scheduler never spoke — a missing scheduler, an admission webhook, or a `schedulerName` nothing
  is servicing. Reachable, and a different problem from every pattern below.
- the scheduler spoke long ago and the record aged out. **This is the likely one for a pod that has
  been Pending for hours**, i.e. exactly when the events are empty.

Use the pod's own `PodScheduled` condition to tell them apart — it is part of the object, so it does
not expire. `PodScheduled=False (Unschedulable: …)` carries the scheduler's verdict and its message
survives the event that first reported it; if the pod has no `PodScheduled` condition at all, the
scheduler genuinely has not processed it. Read the pod's age against the cluster's event retention
before treating an empty section as evidence of anything.

If the message names a specific node, read that node the same way:

```
k8s_inspect(kind: "node", name: "<node>")
```

Reach for `kubectl describe pod <pod> -n <ns>` when you need the pod's affinity rules, tolerations or
volume declarations verbatim.

### 2. Match scheduling failure and investigate

Match the `FailedScheduling` message against the patterns below.

---

#### `Insufficient cpu` / `Insufficient memory` — Not enough resources

No node has enough allocatable CPU or memory to satisfy the pod's resource requests.

Check node resource usage:

```bash
kubectl top nodes
```

Check what the pod is requesting:

```bash
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].resources.requests}'
```

Advise the user to either reduce the pod's resource requests, scale up existing nodes, or add new nodes to the cluster.

---

#### `didn't match Pod's node affinity/selector` — Node affinity/selector mismatch

The pod has a `nodeSelector` or `nodeAffinity` that no available node satisfies.

Check the pod's node selection criteria:

```bash
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.nodeSelector}'
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.affinity}'
```

Check available node labels:

```bash
kubectl get nodes --show-labels
```

Advise the user to either update the pod's selector/affinity or add the required labels to appropriate nodes.

---

#### `had taint` ... `that the pod didn't tolerate` — Taint/toleration mismatch

Nodes have taints that the pod does not tolerate.

Check node taints:

```bash
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'
```

Check the pod's tolerations:

```bash
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.tolerations}'
```

Advise the user to either add the appropriate toleration to the pod or remove the taint from a node.

---

#### `persistentvolumeclaim` ... `not found` / `not bound` — PVC issue

The pod references a PVC that does not exist or is not bound to a PV.

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw