volcano-gang-scheduling

Files (1)

• SKILL.md

SKILL.md (first lines)

# Gang Scheduling Diagnosis

This is a diagnostic guide for Gang scheduling issues in Volcano. Gang scheduling requires that all members of a PodGroup be scheduled simultaneously. If the cluster cannot satisfy the `minMember` requirement, none of the pods will be scheduled.

**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify PodGroups or resource configurations.

## When to Use This Guide

Use this skill when:
- PodGroup status is `Inqueue` but member Pods remain `Pending`
- Events contain `minMember` related errors
- Volcano Job has `minAvailable` or `minMember` that cannot be satisfied
- Some member Pods are running, others are Pending, and the entire group won't start
- You see `FailedScheduling` events mentioning Gang constraints

## Understanding Gang Scheduling

Gang scheduling in Volcano ensures that either all members of a workload are scheduled, or none are. This is crucial for distributed workloads like MPI, TensorFlow, PyTorch where partial scheduling is wasteful.

**Key Concepts:**
- `minMember` (in PodGroup spec): Minimum number of pods that must be scheduled simultaneously
- `minResources` (in PodGroup spec): Aggregate resource floor (e.g., total GPUs) that must be available — **both** `minMember` and `minResources` must be satisfied if set
- `minAvailable` (in Job spec): Similar concept at Job level
- The scheduler checks if there are **simultaneous** resources for all minMember pods before allocating

## Diagnostic Steps

### Step 1: Identify the PodGroup

Find the PodGroup associated with the pending pods:

```bash
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.metadata.annotations.scheduling.volcano.sh/pod-group}'
```

### Step 2: Check PodGroup Status

Get detailed PodGroup information:

```bash
kubectl get podgroup <podgroup-name> -n <namespace> -o yaml
```

**Key fields to examine:**

| Field | Meaning | What to Look For |
|-------|---------|------------------|
| `spec.minMember` | Minimum pods required | Is this number achievable? |
| `spec.minResources` | Aggregate resource floor | Is total cluster capacity sufficient? |
| `status.phase` | Current scheduling phase | Should be `Inqueue` for ready-to-schedule |
| `status.running` | Currently running pods | Compare to minMember |
| `status.pending` | Pending pods | These are waiting for Gang constraint |
| `spec.queue` | Queue name | Check if queue has sufficient resources |

**Common scenarios:**

- `status.phase: Pending` - PodGroup is waiting to be enqueued
- `status.phase: Inqueue` - Ready for scheduling but constraint not met
- `status.running < spec.minMember` - Gang constraint not satisfied

### Step 3: Calculate Resource Requirements

Calculate the total resources needed for the Gang:

```
Total CPU = minMember × single Pod CPU request
Total Memory = minMember × single Pod Memory request
Total GPU = minMember × single Pod GPU request (if applicable)
```

Get a pod's resource requests:

```bash
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources.requests}'
```

### Step 4: Check Cluster Resources

#### Option A: Check Node Resources

View available resources across nodes:

```bash
kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory,GPU:.status.allocatable.nvidia\.com/gpu'
```

Check current resource usage:

```bash
kubectl top nodes
```

#### Option B: Check by Node Labels (if pods have node affinity)

If pods target specific nodes:

```bash
kubectl get nodes -l <label-key>=<label-value> -o wide
```

### Step 5: Check Events for Gang Errors

Look for Gang-specific scheduling errors:

```bash
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name> --sort-by='.lastTimestamp'
```

**Common Gang-related event messages:**

| Message | Meaning | Investigation |
|---------|---------|---------------|
| `minMember not satisfied` | Gang constraint preventing scheduling | Check if total resources >= minMember requirements |
| `gang member not ready` | Some pods in the gang are not ready | Check individual pod status |
| `resource insufficient` | Not enough resources for all members | Use `volcano-resource-insufficient` skill |

### Step 6: Verify Queue Resources

If the PodGroup is in a Queue, check if the queue has sufficient deserved resources:

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw