networkpolicy-debug

Files (1)

• SKILL.md

SKILL.md (first lines)

# NetworkPolicy Connectivity Diagnosis

When pod-to-pod or pod-to-external communication is unexpectedly blocked, and Service/DNS/Ingress diagnostics show no issues, NetworkPolicy is a common root cause. Follow this flow to identify whether a NetworkPolicy is blocking traffic.

**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify or delete NetworkPolicies — that should be left to the user or cluster administrator.

**When to use:** Pod connectivity "suddenly broke" or a newly deployed pod cannot reach other services. Typical clues:
- `service-debug` shows endpoints exist and ports match, but connections time out
- `dns-debug` shows DNS timeouts (may be egress NetworkPolicy blocking UDP 53)
- Traffic works from some pods but not others in the same namespace
- A new NetworkPolicy was recently applied

**Not for other network issues:** If the problem is DNS resolution → use `dns-debug`. If the problem is Service having no endpoints → use `service-debug`. If the problem is Ingress routing → use `ingress-debug`. This skill specifically diagnoses NetworkPolicy-level blocking.

## Key Concepts

- A NetworkPolicy selects pods via `podSelector` and defines allowed `ingress` (incoming) and/or `egress` (outgoing) traffic rules.
- **NetworkPolicy is deny-by-default once applied.** If any NetworkPolicy selects a pod for a given direction (ingress or egress), all traffic in that direction is denied EXCEPT what is explicitly allowed by the rules. Pods with NO NetworkPolicy selecting them allow all traffic.
- Multiple NetworkPolicies selecting the same pod are **additive (union)** — a connection is allowed if ANY matching policy permits it.
- NetworkPolicy requires **CNI support**. If the CNI plugin does not support NetworkPolicy (e.g., Flannel without additional plugins), policies are silently ignored — they can be created but have no effect.
- **`hostNetwork: true` pods are exempt.** Pods using the host network namespace are not selected by any NetworkPolicy — neither as targets nor as sources. A default-deny policy does not protect or restrict hostNetwork pods.

## Diagnostic Flow

### 1. Verify CNI supports NetworkPolicy

Not all CNI plugins enforce NetworkPolicy. If the CNI does not support it, policies are silently ignored — they exist as API objects but have no effect.

Check which CNI is running:

```bash
kubectl get pods -n kube-system -o custom-columns='NAME:.metadata.name' | grep -E 'calico|cilium|weave|antrea|flannel|canal|kube-router'
```

If no results, the CNI may run in a different namespace (e.g., `cilium` in `cilium` namespace, `calico` in `calico-system`). Check other namespaces or inspect the node's CNI config:

```bash
kubectl get pods -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' | grep -E 'calico|cilium|weave|antrea|flannel|canal|kube-router'
```

| CNI | NetworkPolicy support |
|-----|----------------------|
| Calico | Yes |
| Cilium | Yes (also supports extended CiliumNetworkPolicy) |
| Weave Net | Yes |
| Antrea | Yes |
| Canal (Flannel + Calico) | Yes |
| kube-router | Yes |
| Flannel (standalone) | **No** — policies are silently ignored |
| kubenet | **No** |

If the CNI does not support NetworkPolicy:
- Policies exist but do nothing — not the cause of blocked traffic, look elsewhere
- If the user expects policies to work, they need to switch to a CNI that supports them

If the CNI does support NetworkPolicy, continue to step 2.

### 2. List NetworkPolicies in the namespace

```bash
kubectl get networkpolicy -n <ns>
```

If no NetworkPolicies exist in the namespace, standard Kubernetes NetworkPolicy is not the cause — all traffic is allowed by default. However, if the CNI is Calico or Cilium, also check for CNI-specific extended policies that operate independently of standard NetworkPolicy:

```bash
# Cilium extended policies
kubectl get ciliumnetworkpolicy -n <ns> 2>/dev/null
kubectl get ciliumclusterwidenetworkpolicy 2>/dev/null

# Calico extended policies
kubectl get networkpolicy.crd.projectcalico.org -n <ns> 2>/dev/null
kubectl get globalnetworkpolicy.crd.projectcalico.org 2>/dev/null
```

These CNI-specific policies can block traffic even when no standard NetworkPolicy exists, and they take effect at a higher priority. If extended policies exist, examine their rules using `-o yaml`.

If neither standard nor extended policies exist, look elsewhere (firewall rules, service mesh, node-level iptables).

If policies exist (standard or extended), continue to step 3.

### 3. Identify which policies affect the target pod

Kubernetes does not provide a direct API to query "which policies affect this pod." You must manually match each policy's `podSelector` against the pod's labels.

Get the pod's labels:

```bash
kubectl get pod <pod> -n <ns> -o jsonpath='{.metadata.labels}'
```

Get all NetworkPolicies with their full pod selectors:

```bash
kubectl get networkpolicy -n <ns> -o custom-columns='NAME:.metadata.name,SELECTOR:.spec.podSelector'
```

Note: `podSelector` can use both `matchLabels` (exact key-value pairs) and `matchExpressions` (operators like `In`, `NotIn`, `Exists`, `DoesNotExist`). The command above shows both forms. If the output is truncated, use `-o yaml` to see the full selector.

A NetworkPolicy affects the pod if:
- The policy's `podSelector.matchLabels` matches a **subset** of the pod's labels
- The policy's `podSelector.matchExpressions` conditions are satisfied by the pod's labels (e.g., `{key: tier, operator: Exists}` matches any pod with a `tier` label)
- An **empty podSelector** (`{}`) matches ALL pods in the namespace

List the matching policies — these are the ones controlling the pod's traffic.

### 4. Check for default-deny policies

A common pattern is a namespace-wide "deny all" policy:

```bash
kubectl get networkpolicy -n <ns> -o yaml
```

Look for policies with empty ingress or egress rules:

**Default deny all ingress:**
```yaml
spec:

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw