volcano-queue-diagnose

Files (2)

• SKILL.md
• diagnose-queue.sh · scripts/

SKILL.md (first lines)

# Volcano Queue Diagnosis

Diagnose Volcano Queue status, resource allocation, and scheduling bottlenecks. This skill helps understand how resources are distributed across queues and why workloads may be pending due to queue constraints.

**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify queue configurations or delete queues.

**Not applicable to native ResourceQuota:** Volcano Queue and Kubernetes ResourceQuota are independent mechanisms. If the cluster does not use Volcano, use `quota-debug` instead. To check: `kubectl get queue 2>/dev/null` — if it returns an error or empty, Volcano is not installed.

## Usage

```bash
bash skills/core/volcano-queue-diagnose/scripts/diagnose-queue.sh [options]
```

## Parameters

| Parameter | Required | Description |
|-----------|----------|-------------|
| `--queue QUEUE` | no | Queue name to diagnose (default: all queues) |
| `--show-pods` | no | Show pods associated with each queue |
| `--verbose` | no | Show detailed resource breakdown |

## Examples

Diagnose all queues:
```bash
bash skills/core/volcano-queue-diagnose/scripts/diagnose-queue.sh
```

Diagnose specific queue:
```bash
bash skills/core/volcano-queue-diagnose/scripts/diagnose-queue.sh --queue training-queue
```

Show verbose output with pod information:
```bash
bash skills/core/volcano-queue-diagnose/scripts/diagnose-queue.sh --queue training-queue --show-pods --verbose
```

## Understanding Volcano Queues

### Queue Concept

In Volcano, a Queue is a cluster-level resource allocation unit. Jobs and PodGroups are submitted to queues, and the scheduler distributes resources among queues based on:

1. **Weight** - Relative share of cluster resources (proportional: weight 10 vs weight 2 = 83% vs 17%)
2. **Capability** - Maximum resources a queue can use (ceiling, not guarantee — actual allocation depends on cluster capacity and competition)
3. **Parent** - Hierarchical queue relationships (if enabled)

**Important:** A Queue is a **cluster-scoped** resource. PodGroups from **any namespace** can reference the same queue, so cross-namespace resource competition within a queue is expected.

### Queue Status Fields

| Field | Meaning |
|-------|---------|
| `state` | Queue state: Open, Closed, Closing |
| `deserved` | Resources the queue should receive based on weight |
| `allocated` | Resources currently allocated to jobs in this queue |
| `used` | Resources actually used by running pods (≤ allocated) |
| `pending` | Number of PodGroups waiting in the queue |
| `running` | Number of running PodGroups |

## Diagnostic Flow

### Step 1: List All Queues

Get an overview of all queues:

```bash
kubectl get queue
```

**Output columns:**
- NAME: Queue name
- WEIGHT: Queue weight (higher = more resources)
- STATE: Open, Closed, or Closing
- PARENT: Parent queue (for hierarchical queues)

### Step 2: Check Queue Details

Get detailed information about a specific queue:

```bash
kubectl get queue <queue-name> -o yaml
kubectl describe queue <queue-name>
```

**Key sections to examine:**

#### Spec (Configuration)
```yaml
spec:
  weight: 10              # Relative weight (default: 1)
  capability:             # Max resources allowed
    cpu: "100"
    memory: "200Gi"
  reclaimable: true       # Allow resource reclamation
```

#### Status (Runtime State)
```yaml
status:
  state: Open             # Open, Closed, or Closing
  pending: 5              # PodGroups waiting
  running: 10             # Running PodGroups
  deserved:               # Resources this queue should get
    cpu: "40"
    memory: "80Gi"
  allocated:              # Resources actually allocated
    cpu: "35"
    memory: "70Gi"
```

### Step 3: Check Queue Resource Utilization

Calculate utilization ratios:

```
Allocation Ratio = allocated / deserved

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw