statefulset-debug

Files (1)

• SKILL.md

SKILL.md (first lines)

# StatefulSet Rollout & Scaling Failure Diagnosis

When a StatefulSet rollout is stuck, pods are not updating, or scaling is not progressing, follow this flow to identify the root cause.

**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify the StatefulSet, delete pods, or change PVCs — that should be left to the user.

**When to use:** A StatefulSet is not progressing — pods are not updating to the new version, scaling up/down is stuck, or specific ordinal pods are not becoming ready.

**Not for Deployments:** Deployment rollouts have different semantics (parallel, unordered). Use `deployment-rollout-debug` for Deployments.

## Key Concepts

StatefulSets differ fundamentally from Deployments:
- **Fixed pod identity** — pods have stable names with ordinal suffixes (pod-0, pod-1, ...)
- **Ordered operations** — updates go in reverse order (N-1 → 0), scaling up goes in forward order (0 → N-1)
- **Per-pod PVCs** — each pod gets its own PersistentVolumeClaim via `volumeClaimTemplates`
- **Blocking progression** — in OrderedReady mode (default), if pod at ordinal K is not Ready, all pods with ordinal < K will NOT be updated

## Diagnostic Flow

### 1. Get StatefulSet overview

```bash
kubectl get statefulset <name> -n <ns> -o wide
```

Compare the columns:
- **READY** — pods that are running and ready
- **REPLICAS** — desired replica count (from `spec.replicas`)
- **UP-TO-DATE** — pods running the current version (matching `currentRevision` == `updateRevision`)

If `READY < REPLICAS` or there is no `UP-TO-DATE` column showing full count, the rollout or scaling is incomplete.

### 2. Describe the StatefulSet

```bash
kubectl describe statefulset <name> -n <ns>
```

Focus on:
- **Update Strategy** — `RollingUpdate` or `OnDelete`
- **Partition** — if set, only pods with ordinal ≥ partition are updated
- **maxUnavailable** — if set (Kubernetes 1.24+), allows multiple pods to be updated simultaneously instead of one-at-a-time
- **Current Revision / Update Revision** — if different, an update is in progress
- **Events** — look for errors or warnings

### 3. Check pod status by ordinal

First get the StatefulSet's pod selector to reliably find its pods:

```bash
kubectl get statefulset <name> -n <ns> -o jsonpath='{.spec.selector.matchLabels}'
```

Then use the returned labels to list pods:

```bash
kubectl get pods -n <ns> -l <key>=<value> --sort-by='.metadata.name'
```

Identify which ordinal pod is stuck. In a StatefulSet with OrderedReady policy, **the stuck pod blocks all subsequent operations**.

### 4. Match the failure pattern

---

#### OnDelete strategy — Pods not updating after StatefulSet change

The StatefulSet uses `updateStrategy.type: OnDelete`. In this mode, Kubernetes does **not** automatically update pods — the user must manually delete each pod for it to be recreated with the new spec.

```bash
kubectl get statefulset <name> -n <ns> -o jsonpath='{.spec.updateStrategy}'
```

If the output shows `{"type":"OnDelete"}` or no `rollingUpdate` field:

Check if the current and update revisions differ:

```bash
kubectl get statefulset <name> -n <ns> -o jsonpath='current={.status.currentRevision} update={.status.updateRevision}'
```

If they differ, the StatefulSet spec has been updated but pods are still running the old version. This is **expected behavior** for OnDelete — pods must be manually deleted to pick up the new version.

Check which pods are still on the old revision (use the selector from step 3):

```bash
kubectl get pods -n <ns> -l <key>=<value> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.controller-revision-hash}{"\n"}{end}'
```

Pods whose `controller-revision-hash` matches `currentRevision` (not `updateRevision`) are still on the old version.

---

#### RollingUpdate stuck at a specific ordinal — Ordered update blocked

In RollingUpdate mode, StatefulSet updates pods in **reverse ordinal order** (N-1 → N-2 → ... → 0). By default (one-at-a-time), if pod at ordinal K is not Ready, the update stops — pods K-1, K-2, ..., 0 will not be updated.

**Check maxUnavailable** (Kubernetes 1.24+, GA in 1.27):

```bash
kubectl get statefulset <name> -n <ns> -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'
```

If `maxUnavailable` is set (e.g., `3`), multiple pods can be updated simultaneously instead of strict one-at-a-time. In this case, seeing 2-3 pods updating at once is normal — not a sign of being stuck. Only investigate if the number of updating pods is below `maxUnavailable` for an extended period, or if specific pods are stuck in a non-Ready state.

Find pods that are not Ready:

```bash
kubectl get pods -n <ns> -l <key>=<value> --sort-by='.metadata.name'
```

Check the stuck pod's status:
- **Pending** → Use `pod-pending-debug`
- **CrashLoopBackOff / Error** → Use `pod-crash-debug`
- **ImagePullBackOff** → Use `image-pull-debug`
- **Running but not Ready** → Check readiness probe (see below)

If the pod is Running but not Ready:

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw