volcano-job-diagnose

Files (2)

• SKILL.md
• diagnose-job.sh · scripts/

SKILL.md (first lines)

# Volcano Job Diagnosis

Diagnose Volcano Job (batch.volcano.sh/v1beta1) status and issues. This skill checks Job phases, task statuses, PodGroup associations, and overall job health.

**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify job specs or restart jobs — that should be left to the user.

## Usage

```bash
bash skills/core/volcano-job-diagnose/scripts/diagnose-job.sh --job <job-name> --namespace <namespace>
```

## Parameters

| Parameter | Required | Description |
|-----------|----------|-------------|
| `--job JOB` | yes | Job name to diagnose |
| `--namespace NS` | no | Namespace (default: `default`) |
| `--verbose` | no | Show detailed task and pod information |

## Examples

Diagnose a Volcano Job:
```bash
bash skills/core/volcano-job-diagnose/scripts/diagnose-job.sh --job my-training-job --namespace training
```

Verbose mode with task details:
```bash
bash skills/core/volcano-job-diagnose/scripts/diagnose-job.sh --job my-training-job --namespace training --verbose
```

## Understanding Volcano Jobs

### Job Structure

```yaml
apiVersion: batch.volcano.sh/v1beta1
kind: Job
spec:
  schedulerName: volcano
  tasks:
    - name: worker
      replicas: 4
      template:
        spec:
          containers:
            - name: worker
              resources:
                requests:
                  cpu: "4"
                  memory: "8Gi"
  maxRetry: 3              # Max retries before job is Aborted
  policies:
    - event: PodFailed
      action: RestartJob
```

> **Note:** Volcano Jobs can also be queried using the short name `vcjob` (e.g., `kubectl get vcjob`). This is an alias for `job.batch.volcano.sh`. Be careful not to confuse with native Kubernetes `batch/v1 Job` — always use `job.batch.volcano.sh` or `vcjob` for Volcano Jobs.

### Job Phases

| Phase | Meaning |
|-------|---------|
| `Pending` | Job is waiting for resources or admission |
| `Running` | Job is executing |
| `Completing` | Job tasks are completing |
| `Completed` | Job finished successfully |
| `Failed` | Job failed |
| `Restarting` | Job is being restarted due to policy |
| `Terminating` | Job is being terminated |
| `Aborted` | Job was aborted |

### Task Statuses

Each task within a job has its own status:
- `Pending` - Task pods not yet scheduled
- `Running` - Task pods are running
- `Completed` - Task finished
- `Failed` - Task failed

## Diagnostic Flow

### Step 1: Job Overview

Get the Job status:

```bash
kubectl get job.batch.volcano.sh <job-name> -n <namespace> -o yaml
```

**Key fields to check:**
- `status.state.phase` - Current job phase
- `status.failed` - Number of failed tasks
- `status.succeeded` - Number of succeeded tasks
- `status.running` - Number of running tasks
- `status.pending` - Number of pending tasks

### Step 2: Check Tasks

List all tasks and their statuses:

```bash
kubectl get pods -n <namespace> -l volcano.sh/job-name=<job-name> -o wide
```

**What to look for:**
- Pod phases (Pending, Running, Completed, Failed)
- Pod restart counts
- Node assignments

### Step 3: Check PodGroup Association

Find the PodGroup created for this Job:

```bash
kubectl get podgroups -n <namespace> -l volcano.sh/job-name=<job-name>
```

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw