cilium-networking

Files (1)

• SKILL.md

SKILL.md (first lines)

# Cilium networking

## Defaults this skill encodes

1. **CNI = Cilium** (prefer over default flannel-style bootstrap).
2. **LoadBalancer IPs = Cilium LB-IPAM**, not the MetalLB project.
3. **Advertise VIPs on LAN = Cilium L2 Announcement** on host interfaces.
4. Pin fixed VIPs with annotations `lbipam.cilium.io/ips` + `lbipam.cilium.io/ip-pool`.

## Install

Follow current Cilium docs for your k8s/Talos version. Pin Helm chart.

```bash
helm repo add cilium https://helm.cilium.io/
# helm upgrade --install cilium cilium/cilium -n kube-system --version <PIN>
```

Enable features per docs for your version: kube-proxy replacement, LB IPAM, L2 announcements.

## Pool + L2 (required for LAN VIPs)

```yaml
# CiliumLoadBalancerIPPool — DHCP-excluded CIDR on YOUR LAN
# CiliumL2AnnouncementPolicy — interfaces regex matching node NICs
```

See `docs/platform/networking-decisions.md` for full examples.

## Fixed VIP on a Service

```yaml
metadata:
  annotations:
    lbipam.cilium.io/ips: "<VIP>"
    lbipam.cilium.io/ip-pool: servers-pool
spec:
  type: LoadBalancer
```

## Concepts to teach

| Term | Meaning |
|------|---------|
| Pod CIDR | Pod address space (native routing or tunnel) |
| Service CIDR | ClusterIP range |
| LB pool | LAN addresses for type LoadBalancer |
| L2 announce | ARP so LAN hosts reach the VIP |

## Verify

```bash
kubectl -n kube-system get pods -l k8s-app=cilium
kubectl get ciliumloadbalancerippools
kubectl get ciliuml2announcementpolicies
kubectl get svc -A --field-selector spec.type=LoadBalancer
# From another LAN host: ping <VIP>
```

## Footguns

- Exhausting the pool or overlapping DHCP
- Wrong interface regex → no ARP, VIP unreachable
- Accepting bootstrap CNI that replaces Cilium during upgrades
- Calling this MetalLB in GitOps while implementing Cilium (confuses agents)
- **L2 on a NIC without an IP:** connections reset after five idle minutes (the MAC ages out of the switches and frames flood to every node). Add an ARP keepalive DaemonSet on that NIC, or use BGP
- L2 failover after a hard node loss is about 15–20 s with default lease timings; that is expected, not a fault
- VLAN-tagged frames on a Cilium device are dropped unless the VLAN is in `bpf.vlanBypass`
- Multus beside Cilium needs `cni.exclusive: false` **and** `rollOutCiliumPods: true`; otherwise Cilium renames the other CNI's config file and nothing restarts

## BGP instead of L2 (when the gateway can)

- `bgpControlPlane.enabled: true`; `CiliumBGPClusterConfig` + `CiliumBGPPeerConfig` + `CiliumBGPAdvertisement`.
- Timers: hold 3 s, keepalive 1 s. **Graceful restart off** (it keeps a dead node's routes; a hard failure took 23.8 s with it, 2 s without).
- The LB range must not be a network on the gateway. Gateway side: listen range = node network, inbound prefix list = that cluster's LB range, deny-all outbound.
- Measure: curl in a loop, power a node off, read the gap. Record the number.

## Related

- Multus: `skills/multus-secondary-net/`
- Full write-up: `docs/platform/networking-decisions.md`
- L2 vs BGP, measured: `docs/platform/loadbalancer-l2-vs-bgp.md`

From giovannirco/agentic-homelab, MIT.

More from giovannirco/agentic-homelab