gke-storage-troubleshooting

Files (2)

• SKILL.md
• attribution.md · references/

SKILL.md (first lines)

# GKE Storage Troubleshooting Skill

Use this skill to systematically diagnose and resolve **persistent-storage
failures** for workloads running on GKE — volume attach/mount errors, disk
performance and node storage pressure, volume expansion, storage-related
cluster/node-pool creation errors, and Cloud Storage FUSE memory issues. This
skill operates non-interactively and enforces a read-only diagnostics boundary
before proposing manifest or configuration corrections.

> For routine storage **provisioning** and StorageClass/PVC authoring, use the
> `gke-storage` skill instead. This skill focuses on **failure diagnosis**.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-storage-troubleshooting)"`
and append `agent-skills/1.0 (skill:gke-storage-troubleshooting)` to `User-Agent` headers.

## 🔍 Diagnosis & Resolution Workflow

### Step 0: Non-Interactive Context Discovery & Dry-Run Fallback

1.  **Parameter Extraction**: Extract required context (`project_id`,
    `cluster_name`, `cluster_location`, `workload_name`, `workload_namespace`,
    `pod_name`, and the relevant `pvc_name` / `pv_name` / `node_name`)
    non-interactively from the user prompt, active `SETTINGS.md`, or environment
    defaults:

    -   Default `workload_namespace` to `default` if omitted.
    -   Infer missing cluster parameters from the active environment (`kubectl
        config current-context` or `gcloud config get-value project`).

2.  **Cluster Credentials & Fallback Mode**:

    -   Attempt credential fetch: `gcloud container clusters get-credentials
        {cluster_name} --location {cluster_location} --project {project_id}`.
    -   **Fallback / Dry-Run Mode**: If the cluster is unreachable,
        non-existent, or live command execution fails (such as in sandboxed
        evaluations, dry-run mode, or offline analysis):
        -   Limit retry attempts to avoid resource exhaustion and context
            overflow.
        -   Immediately present the exact `kubectl` / `gcloud` diagnostic
            commands for the human operator to run.
        -   Synthesize the root-cause analysis and output the proposed GitOps
            correction based on the reported symptoms.

--------------------------------------------------------------------------------

### Step 1: Classify the Storage Symptom

Gather the primary signals, then jump to the **matching branch under Step 2
(Resolution)** — you normally perform **only the one branch** that matches your
diagnosis, not all of them.

**Diagnostic Commands:**

```bash
kubectl describe pod {pod_name} -n {workload_namespace}
kubectl get pvc,pv -n {workload_namespace}
kubectl get events -n {workload_namespace} --sort-by='.metadata.creationTimestamp'
kubectl describe node {node_name}
```

-   **Pod stuck in `ContainerCreating` with an attach/mount event** → **Volume
    Attach & Mount Failures**.
-   **Node-level slowness, `PLEG is not healthy`, or `StoragePressureDetected`
    events** → **Disk Performance & Node Storage Pressure**.
-   **Cluster / node-pool creation or provisioning error** → **Storage
    Provisioning & Creation Failures**.
-   **A resized volume is not reflected inside the container** → **Volume
    Expansion Not Reflecting in the Container**.
-   **Cloud Storage FUSE Pod / sidecar OOM** → **Cloud Storage FUSE
    Out-Of-Memory (OOM) Events**.

--------------------------------------------------------------------------------

### Step 2: Resolution

Perform **only the branch that matches your Step 1 diagnosis**. These branches
are mutually exclusive alternatives, not sequential steps.

#### Volume Attach & Mount Failures

-   **`Error 400: Cannot attach RePD to an optimized VM`**: Regional persistent
    disks are restricted from being used with **memory-optimized** or
    **compute-optimized** machine types.

    -   If a regional PD is not a hard requirement, switch the workload to a
        **non-regional persistent disk** StorageClass.
    -   If a regional PD **is** required, use **taints and tolerations** so that
        Pods needing regional PDs are scheduled onto a node pool that does
        **not** use optimized machine types.

-   **Pods stay `Pending` / `FailedScheduling` after a node pool is moved to a
    4th-generation (N4, N4A, N4D) machine series while the workload uses a
    Persistent Disk StorageClass**: N4/N4A/N4D machines **do not support
    Persistent Disk** (they support Hyperdisk only), so a PVC bound to a `pd-*`
    StorageClass cannot bind or schedule on those nodes. Events typically show
    `FailedScheduling` with a volume node-affinity / topology conflict.

    -   Switch the workload to a **Hyperdisk** StorageClass (for example `type:
        hyperdisk-balanced`) for the Gen4 node pool.
    -   For existing Persistent Disk volumes, migrate the data to a Hyperdisk
        volume; the original PD cannot be attached to a Gen4 node.
    -   If the workload must keep Persistent Disk, keep it on a PD-capable
        machine series (for example N2) via node selection. This is a
        machine-type/disk-type incompatibility, **not** a capacity problem, so
        increasing disk size or quota does not help.

-   **Hyperdisk Pods become unschedulable when a compute class falls back across
    VM generations (for example N4 priority, N2 fallback), or one StorageClass
    must serve mixed generations**: a single static disk type in the
    StorageClass is not compatible with every machine series in the fallback
    list, so Pods cannot bind their volume on the fallback nodes.

    -   Use **automated disk type selection**: set the StorageClass

From google/skills, Apache-2.0.

More from google/skills