image-pull-debug

Files (1)

• SKILL.md

SKILL.md (first lines)

# Image Pull Failure Diagnosis

When a pod is stuck in `ErrImagePull` or `ImagePullBackOff`, follow this flow to identify the root cause.

**Important:** `ErrImagePull`, `ImagePullBackOff`, and `Back-off pulling image` are NOT causes — they only indicate the pull failed. You MUST proceed through all steps below to find the actual cause. Never conclude with just these status messages.

**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt network-level debugging (ping, curl, iptables, traceroute, etc.) — that is outside the scope of this skill and should be left to the user or network administrator.

## Diagnostic Flow

### 1. Get pod info — one call

```
k8s_inspect(kind: "pod", name: "<pod>", namespace: "<ns>")
```

All three things this step needs come back together: the **waiting reason and message** (which may
already be the root cause), the **image** including its registry prefix, and the **node name** in the
`node` section — which is the argument step 2 needs, and the reason this used to cost three separate
reads before the node was even known.

Also check the image registry:
- If the image has **no registry prefix** (e.g. `nginx:latest`, `envoyproxy/gateway:v1.2.8`), it pulls from **Docker Hub** (`docker.io`).
- If it has a prefix (e.g. `registry.example.com/app:v1`), it pulls from that registry.

### 2. Check containerd logs

Containerd logs are the authoritative source for the root cause. Pod events are often generic ("Failed to pull image") and do not contain the actual error — always check containerd logs.

Use the `node-logs` skill — via `host_script` when the node is a bound SSH host
(check `host_list`), otherwise `node_script`:

```
node_script: node="<nodeName>", skill="node-logs", script="get-node-logs.sh",
             args='--unit containerd --grep-fixed "<image>" --since "1h ago"'
```

Use `--grep-fixed` for the image reference: a registry path is a literal, and its
`.`, `-` and `/` would otherwise be read as regex. Adjust `--since` to cover the
pod's creation time.

If the journal has nothing, try the log files — note the distro difference
(`/var/log/messages` on RHEL/CentOS, `/var/log/syslog` on Ubuntu):

```
node_script: node="<nodeName>", skill="node-logs", script="get-node-logs.sh",
             args='--file /var/log/messages --include-rotated --grep-fixed "<image>"'
```

Read the `status:` line at the end of the output before concluding anything. Only
`no_match` means "read successfully, nothing matched"; `source_error`,
`not_found` and `filter_error` say nothing at all about the image pull. See the
`node-logs` SKILL.md for the full table, and for the tier-3 fallback when neither
SSH nor a debug pod is available on that node.

### 3. Match error and conclude

Match the error from containerd logs (or the `state.waiting.message` from step 1) against the patterns below. Once a pattern matches, **report the root cause to the user and stop**. Do not continue with further diagnostic commands.

If containerd logs have no relevant entries, check events as a supplementary source:

```bash
kubectl get events -n <ns> --field-selector involvedObject.name=<pod>
```

If still no match, report whatever error information you have found and let the user decide next steps. Do NOT start autonomous network investigation.

---

#### `not found` / `manifest unknown` — Image does not exist

The image name or tag does not exist in the registry. Inform the user to verify the image name and tag.

---

#### `unauthorized` / `access denied` / `denied` — Authentication failed

The registry rejected the request. Advise the user to:
1. Configure an imagePullSecret for the pod with valid registry credentials
2. Or adjust image permissions on the registry side to allow access

---

#### `x509` / `certificate` / `tls` — Certificate not trusted

The node's containerd does not trust the registry's CA certificate. Advise the user to add the registry CA to the node's containerd trust config (`/etc/containerd/certs.d/`) or system trust store.

---

#### `no such host` / `lookup.*failed` — DNS resolution failed

The registry hostname cannot be resolved. Advise the user to check the hostname spelling and node DNS config.

---

#### `connection reset by peer` — Remote reset

TCP reached the registry but was reset by the remote end — a server-side or intermediary issue.

---

#### `i/o timeout` / `dial tcp.*timeout` — Connection timed out

The node cannot establish a TCP connection to the registry. Common causes: firewall blocking, proxy misconfiguration, or registry unreachable from the node's network.

**Docker Hub specific:** If the image is from Docker Hub (`docker.io`) and the node is in mainland China, this is almost certainly caused by network restrictions (GFW). Advise the user to use a registry mirror or re-tag the image to a domestically accessible registry.

Report the timeout to the user and stop. Do NOT attempt network diagnostics (ping, curl, iptables, etc.).

---

#### `connection refused` — Connection refused

TCP reached the host but the port is not listening. The registry service is down or on a different port.

---

#### `too many requests` / `429` / `rate limit` — Rate limited

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw