Triage Kubernetes Node Pressure

Files (1)

• SKILL.md

SKILL.md (first lines)

## When to use

Use when nodes show `MemoryPressure`, `DiskPressure`, or pods are evicted due to node conditions.

## Preconditions

- You have read access to node and pod data.
- You can identify the affected node(s) or the evicted pods.

## Procedure

1. Confirm the node condition and the time it started.
2. Identify impacted workloads (evictions, pending pods, restarts).
3. Determine whether the issue is a single node or widespread.
4. Identify top contributors (largest pods, runaway logs, tmpfs, imagefs).
5. Choose the lowest-risk mitigation that restores capacity.

## Decision points

- Single node only: consider cordon/drain after identifying a safe target.
- Many nodes: treat as capacity or systemic issue (autoscaling, noisy neighbor policy).
- Disk pressure: check image garbage, log growth, emptyDir usage.
- Memory pressure: check memory limits, leaks, and node size.

## Verification

- Node condition clears and stays stable.
- Evictions stop and pending pods schedule.
- Workload SLO signals return to baseline.

## Rollback / undo

- If you drained a node and it increases impact, stop draining and rebalance workloads.
- If you changed resource requests/limits, revert to previous values.

## Escalation

- Platform team for cluster-wide pressure or autoscaler issues.
- Service owners for runaway resource usage.

## Examples

```bash
kubectl describe node <node>
kubectl get pods -A -o wide | grep <node>
kubectl get events -A --sort-by=.lastTimestamp | tail -n 50
```

From bregman-arie/devops-sre-skills, Apache-2.0.

More from bregman-arie/devops-sre-skills