service-debug

Files (1)

• SKILL.md

SKILL.md (first lines)

# Service Connectivity Diagnosis

When a Service is unreachable, returns connection refused, or has no backends, follow this flow to identify the root cause.

**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify the Service, Endpoints, or pod specs — that should be left to the user.

## Diagnostic Flow

### 1. Check the Service

```bash
kubectl get svc <service> -n <ns> -o wide
```

Confirm the Service exists and note:
- **TYPE** — ClusterIP, NodePort, LoadBalancer, or ExternalName
- **CLUSTER-IP** — should not be `None` unless it is a headless service
- **PORT(S)** — the port(s) the Service exposes
- **SELECTOR** — the label selector used to find backend pods

### 2. Check Endpoints

```bash
kubectl get endpoints <service> -n <ns>
```

- If `ENDPOINTS` column shows `<none>` or is empty, no pods match the Service's selector — go to step 3.
- If endpoints exist, note the IPs and ports — go to step 4.

For more detail (including `notReadyAddresses`):

```bash
kubectl describe endpoints <service> -n <ns>
```

### 3. Investigate selector and pods

The Service's selector must match labels on running, ready pods.

Get the Service selector:

```bash
kubectl get svc <service> -n <ns> -o jsonpath='{.spec.selector}'
```

List pods that match the selector:

```bash
kubectl get pods -n <ns> -l <key>=<value> -o wide
```

(Replace `<key>=<value>` with each selector pair from the previous command.)

If no pods match:
- The selector may be wrong (typo, outdated label)
- The pods may be in a different namespace
- The pods may not exist yet

If pods exist but none are `Ready`:
- Pods may be starting, crashing, or failing readiness probes
- Use `pod-crash-debug` or check readiness probes

### 4. Check port configuration

Verify that the Service port, targetPort, and the container's actual listening port all align.

```bash
kubectl get svc <service> -n <ns> -o jsonpath='{range .spec.ports[*]}port={.port} targetPort={.targetPort} protocol={.protocol}{"\n"}{end}'
```

```bash
kubectl get pod <backend-pod> -n <ns> -o jsonpath='{range .spec.containers[*].ports[*]}containerPort={.containerPort} protocol={.protocol}{"\n"}{end}'
```

The Service's `targetPort` must match one of the container's `containerPort` values (or the named port). If they don't match, traffic will be sent to the wrong port.

Note: if the container does not declare `containerPort`, the traffic may still work — `containerPort` is informational. The real check is whether the application is actually listening on the `targetPort`.

### 5. Match patterns and conclude

---

#### Endpoints empty — No pods match selector

The Service has no endpoints because no running, ready pods match its label selector.

Possible causes:
- **Selector mismatch** — the Service selector labels don't match the pod labels (typo or outdated after a rename)
- **No pods exist** — the backing Deployment/StatefulSet has 0 replicas or hasn't been created
- **Pods not ready** — pods exist but are failing readiness probes, so they are excluded from Endpoints

Advise the user to compare the Service selector with the pod labels and ensure pods are running and ready.

---

#### `Connection refused` — Port mismatch or application not listening

Endpoints exist, but connections to the Service are refused.

- The Service `targetPort` may not match the port the application is actually listening on
- The application may not have started yet or may have crashed

Check the application logs:

```bash
kubectl logs <backend-pod> -n <ns> --tail=50
```

---

#### Service type `ExternalName` not resolving — DNS-based service issue

ExternalName services return a CNAME record. If the target hostname is unreachable or doesn't resolve, the service will appear broken.

```bash
kubectl get svc <service> -n <ns> -o jsonpath='{.spec.externalName}'
```

Verify the external name resolves from within the cluster (see `dns-debug` skill).

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw