gke-ai-troubleshooting-jobset-interruption

Files (4)

• SKILL.md
• attribution.md · references/
• failure_signatures.md · references/
• validate_queries.sh · scripts/

SKILL.md (first lines)

# GKE JobSet Interruption Troubleshooting

Use this skill to systematically diagnose and resolve JobSet interruptions,
restarts, and preemptions on GKE clusters hosting large-scale AI/ML workloads.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-jobset-interruption)"`
and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-jobset-interruption)` to `User-Agent` headers.

## ⚠️ Prerequisites & Sandbox Rules

-   JobSet metrics package must be enabled in `kube-state-metrics` for your
    cluster.
-   Cloud Logging and Cloud Monitoring enabled for the Google Cloud Project.
-   **Autonomous & Sandbox Execution Rule**: If API queries, tools, or commands
    encounter `403 Permission Denied`, authentication errors, or network
    isolation, **do NOT enter authentication or credential troubleshooting
    loops**. Populate the query templates with the acquired variables
    (`{project_id}`, `{cluster_name}`, `{workload_name}`, `{start_time}`,
    `{end_time}`), inspect any locally staged telemetry or mock data files if
    available, and complete the diagnostic workflow and resolution
    recommendations autonomously.

## 🔍 Diagnostic Workflow

### Step 0: Context Acquisition & Time Window Definition

Independently gather context using tools, workspace files, environment details,
or user prompt context:

-   **Project ID** (`{project_id}`)
-   **Cluster Name** (`{cluster_name}`)
-   **Workload Name (JobSet Name)** (`{workload_name}`)
-   **Workload Namespace** (`{namespace}`)
-   **Issue Time** (`{issue_time}`)

If specific variables are not explicitly provided by the user, inspect cluster
resources or logs to determine them, or use the `{variable}` placeholders
provided.

#### Time Handling Rules

1.  **Autonomous Time Window**: If a relative time (e.g., "X minutes ago") or no
    exact timestamp is provided, calculate the query window based on current
    time or available log timestamps.
2.  **Window Calculation**: If a timestamp `{issue_time}` is available (or
    calculated as `T`), set `{start_time}` = `T - 30m` and `{end_time}` = `T +
    30m`.

--------------------------------------------------------------------------------

### Step 1: Identify JobSet Restarts and Attempts [Low Risk]

Verify if the JobSet is experiencing restart loops and determine the frequency
of restarts.

#### Visual Chart / MQL Query - restarts

-   **MQL Query Specification**:

    ```mql
    fetch prometheus_target
    | metric 'prometheus.googleapis.com/kube_jobset_restarts/gauge'
    | filter resource.cluster_name == '{cluster_name}' && metric.jobset_name == '{workload_name}'
    | align next_older(1m)
    | every 1m
    | group_by [metric.jobset_name], [val: max(value)]
    ```

#### PromQL Metric Query - restarts

-   **PromQL Query Specification**:

    ```promql
    kube_jobset_restarts{jobset_name="{workload_name}", cluster="{cluster_name}"}
    ```

-   **Diagnostic Logic**: A non-zero or increasing value for restarts indicates
    that the JobSet is being actively restarted by the controller due to worker
    failure or interruption.

-   **Automation**: Proceed to Step 2 automatically after reporting findings.

--------------------------------------------------------------------------------

### Step 2: Inspect Nodepool Interruptions [Low Risk]

Determine if the JobSet restarts were triggered by physical nodepool-level
events (such as spot preemptions, maintenance, or host terminations).

#### A. Metrics Query (Nodepool Interruption Counts)

##### Visual Chart / MQL Query - interruptions

-   **MQL Query Specification**:

    ```mql
    fetch k8s_node_pool
    | metric 'kubernetes.io/node_pool/interruption_count'
    | filter cluster_name == '{cluster_name}'
    | align next_older(10m)
    | every 10m
    | group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]
    ```

##### PromQL Query - interruptions

-   **PromQL Query Specification**:

    ```promql
    sum by (interruption_type, interruption_reason, node_pool_name, cluster_name) (
      avg_over_time(kubernetes_io:node_pool_interruption_count{cluster_name="{cluster_name}"}[10m])
    )
    ```

From google/skills, Apache-2.0.

More from google/skills