gke-workload-troubleshooting

Files (2)

• SKILL.md
• attribution.md · references/

SKILL.md (first lines)

# GKE Workload Troubleshooting Skill

Use this skill to systematically diagnose and resolve failures in application
workloads deployed in GKE clusters. This skill operates non-interactively and
enforces a read-only diagnostics boundary: it only **proposes** fixes — whether
Kubernetes manifest/config patches or Google Cloud changes (for example `gcloud`
IAM bindings or node-pool recreation) — and never executes live mutations
itself.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-workload-troubleshooting)"`
and append `agent-skills/1.0 (skill:gke-workload-troubleshooting)` to `User-Agent` headers.

## 🔍 Diagnostic Workflow

### Step 0: Non-Interactive Context Discovery & Time Window Definition

1.  **Parameter Extraction**: Extract required context (`project_id`,
    `cluster_name`, `cluster_location`, `workload_name`, `workload_namespace`)
    non-interactively from the user prompt, active `SETTINGS.md`, or active
    environment defaults:

    -   Default `workload_namespace` to `default` if omitted.
    -   Infer missing cluster parameters from active environment (`kubectl
        config current-context` or `gcloud config get-value project`).
    -   Prioritize non-interactive context discovery from prompts and
        environment defaults to ensure autonomous execution flow.

2.  **Cluster Credentials & Fallback Mode**:

    -   Attempt credential fetch: `gcloud container clusters get-credentials
        {cluster_name} --region/--zone {cluster_location}`
    -   **Fallback / Dry-Run Mode**: If the cluster is unreachable,
        non-existent, or live command execution fails (such as in sandboxed
        evaluations, dry-run mode, or offline analysis):
        -   Limit retry attempts to avoid resource exhaustion and context
            overflow in unreachable cluster scenarios.
        -   Immediately present the exact sequence of `kubectl` diagnostic
            commands for the human operator to run.
        -   Synthesize the root cause analysis and output the proposed GitOps
            manifest fix based on the reported symptoms.

3.  **Time Handling & Fallbacks**:

    -   **Determine Issue Timestamp ({issue_time})**:
        -   **Specific Time Provided**: If the user provides a specific
            timestamp, use it as `{issue_time}`.
        -   **Relative Time Provided (e.g., "5 minutes ago")**: Dynamically
            calculate the corresponding UTC timestamp based on current system
            time, and use it as `{issue_time}`.
        -   **No Time Provided (Default)**: Use current system time as
            `{issue_time}`.
    -   **Window Calculation**: Center a 1-hour query window around
        `{issue_time}` (`start_time` = `{issue_time} - 30m`, `end_time` =
        `{issue_time} + 30m`).

--------------------------------------------------------------------------------

### Step 1: Analyze Pod Status and Conditions

Inspect the workload's active pod states and controller status.

**Diagnostic Commands:**

```bash
# 1. Inspect the deployment's actual selector labels:
kubectl get deployment {workload_name} -n {workload_namespace} -o jsonpath='{.spec.selector.matchLabels}'
# 2. Query the pods using the returned labels, for example:
kubectl get pods -l {selector_labels} -n {workload_namespace}
kubectl get deploy/{workload_name} -n {workload_namespace} -o yaml
```

#### Diagnostic Decision Tree:

-   **Phase: Pending**:
    -   The Pod cannot schedule on any node. Proceed directly to **Step 2 (Query
        Namespace Events)**.
-   **State: CrashLoopBackOff / Error**:

    -   The container boots but exits repeatedly; the `kubelet` restarts it with
        an increasing back-off delay of up to five minutes. First read the
        terminated **reason** and **exit code**:

    ```bash
    kubectl describe pod {pod_name} -n {workload_namespace}
    kubectl get pod {pod_name} -n {workload_namespace} -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'
    ```

    -   **Reason: OOMKilled (Exit Code 137)**: The container's memory limit was
        reached. Proceed to **Step 3 (Inspect Logs) → OOM Analysis** to classify
        container-level vs node-level, then **Step 5** to propose fixes.
    -   **Exit Code 0 (successful exit)**: Unexpected for a long-running
        Deployment/StatefulSet — `restartPolicy: Always` restarts the finished
        process, creating the loop. Common causes: the `command`/`entrypoint`
        does not start a persistent process, a worker exits on an empty queue,
        or a missing/invalid config (e.g., an unattached or mis-keyed
        `ConfigMap` volume) makes the app exit cleanly. Proceed to **Step 3
        (Inspect Logs)**.
    -   **Exit Code 128**: Invalid `command`/`entrypoint` — the executable path
        is wrong or absent in the image. Verify the container command in the
        manifest.
    -   **Exit Code 1 or other non-zero**: The application crashed —
        configuration errors, missing/invalid env vars or config files,
        unreachable dependencies, or auth failures (`401`/`403`) on Google Cloud
        calls (check the Pod's IAM / Workload Identity Federation). Proceed
        directly to **Step 3 (Inspect Logs)**.
    -   If the exit code looks healthy but the container keeps restarting,
        suspect a **liveness probe failure** (see Step 3).

-   **State: ImagePullBackOff / ErrImagePull**:

    -   The kubelet cannot pull the container image. `ImagePullBackOff` means it
        keeps retrying with back-off; `ErrImagePull` is a general,
        non-recoverable pull error. Related statuses: `InvalidImageName`,

From google/skills, Apache-2.0.

More from google/skills