pod-crash-debug

Files (1)

• SKILL.md

SKILL.md (first lines)

# Pod Crash Failure Diagnosis

When a pod is stuck in `CrashLoopBackOff`, `Error`, `OOMKilled`, or `RunContainerError`, follow this flow to identify the root cause.

**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to fix the application code or modify resource limits — that should be left to the user.

## Diagnostic Flow

### 1. Read the pod and its surroundings — one call

```
k8s_inspect(kind: "pod", name: "<pod>", namespace: "<ns>")
```

This returns, in a single call, everything the first two steps of this flow used to take three or
four: the phase, the per-container readiness and restart count, the **last termination reason and
exit code**, the pod's recent events, the state of the **node** it runs on, and its **owner**'s
replica counts.

What to read:
- `last: OOMKilled exit 137` → memory limit; go to §"OOMKilled" below.
- `last: Error exit <n>` → the application exited; step 2 (previous logs) is the next move.
- `waiting: <reason>` → it never started. `ImagePullBackOff` / `ErrImagePull` belong to the
  `image-pull-debug` skill, `CreateContainerConfigError` to a missing ConfigMap or Secret.
- The `node` section — a node that is `Ready=False`, cordoned, or under `MemoryPressure` explains a
  crash that has nothing to do with the pod.
- The final `status:` line. `partial (node: forbidden)` means that section is **missing**, not empty,
  so do not conclude the node is healthy; an empty events section means the pod genuinely has none.

Reach for `kubectl describe pod <pod> -n <ns>` only when you need something the summary does not
carry — volume mounts, the full probe configuration, or the complete event history.

### 2. Get previous container logs

```bash
kubectl logs <pod> -n <ns> --previous --tail=200
```

If the pod has multiple containers, specify the crashing container:

```bash
kubectl logs <pod> -n <ns> -c <container> --previous --tail=200
```

If `--previous` fails with "previous terminated container not found", try current logs:

```bash
kubectl logs <pod> -n <ns> --tail=200
```

### 3. Match error and conclude

Match the information from steps 1-2 against the patterns below. Once a pattern matches, **report the root cause to the user and stop**.

---

#### `OOMKilled` / exit code 137 (from OOM) — Out of Memory

The container exceeded its memory limit and was killed by the kernel OOM killer.

Check the container's resource limits:

```bash
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].resources}'
```

Advise the user to either increase the memory limit or investigate the application's memory usage (possible memory leak).

---

#### Exit code 137 (no OOMKilled) — SIGKILL

The container was killed by SIGKILL but not due to OOM. Common causes:
- Liveness probe failure — check `kubectl describe pod` for `Unhealthy` events with `Liveness probe failed`
- Manual kill or preemption

If liveness probe failures are present, advise the user to adjust probe timing (`initialDelaySeconds`, `timeoutSeconds`, `periodSeconds`) or fix the health endpoint.

---

#### Exit code 1 — Application error

The application exited with a generic error code. The root cause is in the container logs from step 3. Report the relevant error lines to the user.

---

#### Exit code 2 — Shell/binary misuse

Often indicates a missing binary, incorrect command syntax, or shell script error. Check the container's `command` and `args`:

```bash
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[0].command} {.spec.containers[0].args}'
```

---

#### Exit code 126 — Permission denied

The entrypoint binary exists but is not executable. Advise the user to check file permissions in the container image.

---

#### Exit code 127 — Command not found

The entrypoint binary does not exist in the container image. Advise the user to verify the image contains the expected binary and the `command` field is correct.

---

#### `RunContainerError` — Container failed to start

The container runtime failed to start the container. Common causes:
- Volume mount errors (invalid path, read-only filesystem)
- ConfigMap/Secret not found
- Invalid security context

Check events from step 2 for the specific error message and report it to the user.

---

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw