volcano-scheduler-logs

Files (2)

• SKILL.md
• get-scheduler-logs.sh · scripts/

SKILL.md (first lines)

# Volcano Scheduler Logs

Retrieve and analyze Volcano scheduler logs to understand scheduling decisions, failures, and performance issues.

**Scope:** This skill is for **diagnosis only**. It retrieves logs for analysis but does not modify any cluster state.

## Usage

```bash
bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh [options]
```

## Parameters

| Parameter | Required | Description |
|-----------|----------|-------------|
| `--keyword KEYWORD` | no | Filter logs by keyword (case-insensitive) |
| `--pod POD` | no | Filter logs related to specific pod name |
| `--since TIME` | no | Show logs newer than relative time (e.g., 10m, 1h) |
| `--lines N` | no | Number of lines to show (default: 100) |
| `--follow` | no | Stream logs in real-time (Ctrl+C to stop) |
| `--previous` | no | Show logs from previous container instance (after restart) |

## Examples

Get recent scheduler logs:
```bash
bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh
```

Search for error messages:
```bash
bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --keyword error
```

Get logs for a specific pod:
```bash
bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --pod my-job-0
```

Get last 500 lines from the past hour:
```bash
bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --since 1h --lines 500
```

Stream logs for gang scheduling issues:
```bash
bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --keyword gang --follow
```

Check logs from previous scheduler instance (after crash/restart):
```bash
bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --previous --lines 200
```

## Common Keywords for Filtering

| Keyword | Use Case |
|---------|----------|
| `error` | Find error messages and failures |
| `FailedScheduling` | Scheduling failures |
| `allocate` | Resource allocation attempts |
| `gang` | Gang scheduling decisions |
| `minMember` | MinMember constraint issues |
| `preempt` | Preemption events |
| `reclaim` | Resource reclamation |
| `enqueue` | Queue admission decisions |
| `bind` | Pod binding attempts |
| `queue` | Queue-related decisions |
| `proportion` | Proportion plugin decisions |
| `priority` | Priority-related decisions |

## Understanding Scheduler Logs

### Log Format

Volcano scheduler logs typically follow this format:
```
I0102 15:04:05.123456       1 scheduler.go:123] Starting scheduling session
I0102 15:04:05.234567       1 allocate.go:456] Try to allocate resources for Job <namespace>/<job-name>
E0102 15:04:05.345678       1 gang.go:789] Failed to schedule pod <pod-name>: minMember not satisfied
```

**Log levels:**
- `I` - Info: Normal operation information
- `W` - Warning: Unusual but non-fatal conditions
- `E` - Error: Failures and errors
- `F` - Fatal: Critical errors causing shutdown

### Common Log Patterns

#### Session Start
```
Starting scheduling session
Starting scheduling loop
```
- Indicates scheduler is processing a new batch of pending pods

#### Enqueue Decisions
```
Try to enqueue pod group
PodGroup <name> is enqueued
PodGroup <name> is pending
```
- Shows whether pod groups are admitted to the queue

#### Allocation Attempts
```
Try to allocate resources for Job
Try to allocate for task
```
- Shows scheduling attempts for specific jobs/pods

#### Gang Scheduling
```
minMember not satisfied
gang member not ready
Waiting for gang members
```

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw