volcano-resource-insufficient

Files (1)

• SKILL.md

SKILL.md (first lines)

# Resource Insufficiency Diagnosis

This guide helps diagnose resource shortage issues in Volcano-scheduled workloads. Resource insufficiency is one of the most common causes of scheduling failures.

**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify resource quotas or delete workloads.

## When to Use This Guide

Use this skill when:
- Events show `Insufficient cpu` or `Insufficient memory`
- Pods are stuck in `Pending` with resource-related events
- Nodes show zero allocatable resources
- Pods are being `OOMKilled` (Out of Memory)
- `FailedScheduling` events mention resource constraints

## Types of Resource Issues

### 1. Cluster-Wide Resource Exhaustion
The entire cluster lacks sufficient resources to meet the workload demands.

### 2. Resource Fragmentation
Total resources exist but are distributed across too many nodes to satisfy specific scheduling constraints (like Gang scheduling).

### 3. Per-Node Resource Shortage
Individual nodes lack enough resources, even though the cluster as a whole has capacity.

### 4. Queue Resource Limits
The Queue has reached its deserved resource limit, preventing new pods from being scheduled.

## Diagnostic Steps

### Step 1: Identify Resource Shortage Type

Check the specific error message in events:

```bash
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name> --sort-by='.lastTimestamp'
```

**Common error patterns:**

| Error Message | Resource Type | Scope |
|--------------|---------------|-------|
| `Insufficient cpu` | CPU | Node-level |
| `Insufficient memory` | Memory | Node-level |
| `Insufficient nvidia.com/gpu` | GPU | Node-level |
| `0/N nodes are available` | General | Cluster-level |
| `exceeded quota` | Queue-level | Queue limit |

### Step 2: Check Pod Resource Requests

Determine how much resources the pod is requesting:

```bash
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources.requests}'
```

For detailed breakdown:
```bash
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A 10 "resources:"
```

**Key fields:**
- `resources.requests.cpu` - CPU cores requested (e.g., "100m" = 0.1 core, "2" = 2 cores)
- `resources.requests.memory` - Memory requested (e.g., "1Gi", "512Mi")
- `resources.requests.nvidia.com/gpu` - GPUs requested
- `resources.limits` - Maximum allowed (may be different from requests)

### Step 3: Check Node Allocatable Resources

View total allocatable resources per node:

```bash
kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory,GPU:.status.allocatable.nvidia\.com/gpu,PODS:.status.allocatable.pods'
```

For detailed node information:
```bash
kubectl describe node <node-name>
```

**Key concepts:**
- `allocatable` = Total capacity - System reserved - Kubelet reserved
- `capacity` = Total hardware capacity
- The difference is reserved for system/Kubernetes daemons

### Step 4: Check Current Resource Usage

If metrics-server is available:

```bash
kubectl top nodes
```

For per-node pod usage:
```bash
kubectl top pods --all-namespaces --sort-by=cpu | head -20
kubectl top pods --all-namespaces --sort-by=memory | head -20
```

**Note:** If metrics-server is not available, you can still see resource allocation (requests) but not actual usage.

### Step 5: Calculate Resource Availability

For each node, calculate available resources:

```
Available CPU = allocatable.cpu - sum(all pod requests on node)
Available Memory = allocatable.memory - sum(all pod requests on node)
```

Quick check with:
```bash
kubectl describe node <node-name> | grep -A 20 "Allocated resources"
```

**Look for:**
- `cpu-requests` vs `cpu-capacity`
- `memory-requests` vs `memory-capacity`

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw