gke-workload-scaling-troubleshooting

Files (2)

• SKILL.md
• attribution.md · references/

SKILL.md (first lines)

# GKE Workload Scaling Troubleshooting Skill

Use this skill to systematically diagnose and resolve **HorizontalPodAutoscaler
(HPA) failures** on GKE — metrics reported as `<unknown>`, `FailedGet*` events,
missing resource requests, custom/external metrics-pipeline breakage, HPA that
refuses to scale up or down, scale-to/from-zero issues, and slow HPA reaction on
large clusters. This skill operates non-interactively and enforces a read-only
diagnostics boundary before proposing manifest or configuration corrections.

> For configuring HPA/VPA objects and scaling **best practices**, use the
> `gke-workload-scaling` skill instead. This skill focuses on **failure
> diagnosis**.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-workload-scaling-troubleshooting)"`
and append `agent-skills/1.0 (skill:gke-workload-scaling-troubleshooting)` to `User-Agent` headers.

## 🔍 Diagnosis & Resolution Workflow

### Step 0: Non-Interactive Context Discovery & Dry-Run Fallback

1.  **Parameter Extraction**: Extract required context (`project_id`,
    `cluster_name`, `cluster_location`, `hpa_name`, `workload_name`,
    `workload_namespace`) non-interactively from the user prompt, active
    `SETTINGS.md`, or environment defaults:

    -   Default `workload_namespace` to `default` if omitted.
    -   Infer missing cluster parameters from the active environment (`kubectl
        config current-context` or `gcloud config get-value project`).

2.  **Cluster Credentials & Fallback Mode**:

    -   Attempt credential fetch: `gcloud container clusters get-credentials
        {cluster_name} --location {cluster_location} --project {project_id}`.
    -   **Fallback / Dry-Run Mode**: If the cluster is unreachable,
        non-existent, or live command execution fails (such as in sandboxed
        evaluations, dry-run mode, or offline analysis):
        -   Limit retry attempts to avoid resource exhaustion and context
            overflow.
        -   Immediately present the exact `kubectl` / `gcloud` diagnostic
            commands for the human operator to run.
        -   Synthesize the root-cause analysis and output the proposed GitOps
            correction based on the reported symptoms.

--------------------------------------------------------------------------------

### Step 1: Inspect the HPA and Classify the Symptom

Start every investigation with `kubectl describe hpa`, then route to the matching
branch. The three key sections are **`Metrics`** (an `<unknown>` value means the
HPA hasn't fetched the metric or the pipeline is broken), **`Conditions`**
(`AbleToScale`, `ScalingActive`, `ScalingLimited` — a `False` status marks a
failure), and **`Events`** (specific reasons such as `FailedGetScale` or
`FailedGetResourceMetric`).

**Diagnostic Commands:**

```bash
kubectl describe hpa {hpa_name} -n {workload_namespace}
kubectl get hpa {hpa_name} -n {workload_namespace} -o yaml
```

For historical events, query Cloud Logging (the HPA events survive after the live
`Events` list rolls over):

```
resource.type="k8s_cluster"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
logName="projects/{project_id}/logs/events"
jsonPayload.involvedObject.kind="HorizontalPodAutoscaler"
```

Route by signal:

-   **`FailedGetScale`, `FailedComputeMetricsReplicas`, `Error 400 ... label
    is not allowed`, or fluctuating replicas from competing HPAs** → **Branch A**
    (Configuration Errors).
-   **`FailedGetResourceMetric`, `unable to fetch pod metrics`, or `multiple
    services selecting the same target`** → **Branch B** (Workload & Service
    Errors).
-   **`<unknown>` custom/external metric, `FailedGetExternalMetric` /
    `FailedGetCustomMetric`, or `no known available metric versions found`** →
    **Branch C** (Metrics API & Data Availability).
-   **Conditions all `True` / no errors but the workload won't scale up or down**
    → **Branch D** (Healthy but Unexpected Scaling).
-   **Workload configured with `minReplicas: 0` won't scale to or from zero** →
    **Branch E** (Scale To / From Zero).
-   **Correct HPA but slow reaction on a cluster with many HPA objects** →
    **Branch F** (Slow Recalculation on Large Clusters).

--------------------------------------------------------------------------------

### Step 2: Resolution — Route to the Matching Branch

Based on the signal you classified in Step 1, jump to **one** of the
mutually-exclusive branches below (A–F). These are alternatives — you do **not**
run them in sequence. After applying the branch's fix, go to Step 3 to present it
as a reviewable GitOps change.

#### Branch A: HorizontalPodAutoscaler Configuration Errors

-   **`FailedGetScale` — `unable to get the target's current scale: ... "TARGET"
    not found`**: the `scaleTargetRef` doesn't resolve to an existing scalable
    workload.

    -   Verify the `scaleTargetRef` `name`, `kind`, and `apiVersion` exactly
        match the target workload's metadata.
    -   Confirm the target workload exists **in the same namespace** as the HPA
        (a missing `-n` puts objects in `default`, causing a mismatch).
    -   The target must be a scalable kind (Deployment, StatefulSet,
        ReplicaSet) — you **cannot** autoscale a DaemonSet.

-   **`FailedComputeMetricsReplicas` — `invalid metrics (1 invalid out of 1)`**:

From google/skills, Apache-2.0.

More from google/skills