gke-workload-identity

Files (2)

• SKILL.md
• attribution.md · references/

SKILL.md (first lines)

# GKE Workload Identity Federation Troubleshooting Skill

Use this skill to systematically diagnose why a Pod using **Workload Identity
Federation for GKE** cannot authenticate to Google Cloud APIs. Typical symptoms:

-   `HTTP/403 ... Permission 'iam.serviceAccounts.getAccessToken' denied on
    resource`
-   `google.auth.exceptions ... could not find default credentials` /
    `ComputeEngineCredentials cannot find the metadata server`
-   API calls that unexpectedly use the node's default Compute Engine service
    account instead of the workload's identity.

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-workload-identity)"`
and append `agent-skills/1.0 (skill:gke-workload-identity)` to `User-Agent` headers.

## Read-only boundary

This skill is **diagnostic and non-interactive**. It only **reads** cluster,
IAM, and logging state and **proposes** fixes as commands or GitOps manifest
changes for a human to apply. It must **never** create or modify IAM bindings,
KSA annotations, node pools, or clusters automatically. When evidence is missing
or the fix requires a privileged change, summarize findings and hand off to a
human (see Step 7).

## Output discipline (apply to every conclusion)

When you report a diagnosis, always:

1.  **Name the single most-likely root cause** (not an open-ended list of
    possibilities).
2.  **Explicitly rule out** the other plausible causes, citing the evidence that
    excludes them. In particular, when the cause is node-pool configuration,
    state plainly that it is **not** a KSA annotation or IAM binding problem;
    when the cause is a missing role on a target resource, state that Workload
    Identity itself is **not** misconfigured.
3.  **Give the exact remediation as a proposed change for a human** — a concrete
    `gcloud` command (or GitOps manifest edit) with the real identifiers filled
    in — and never apply it automatically.

--------------------------------------------------------------------------------

## Diagnostic Workflow

### Step 0: Context discovery & time window

Collect (from the user or the failing resource): `PROJECT_ID`, `PROJECT_NUMBER`,
`CLUSTER`, cluster `LOCATION`, `NAMESPACE`, the **KSA** the Pod runs as, the
**node and node pool** the Pod is scheduled on (used in Step 2), the **target
resource / API** being called, and the **exact error string**. Define a time
window around the first observed failure for log queries.

```bash
# Resolve the project number (used in the direct-binding principal identifier).
gcloud projects describe "{PROJECT_ID}" --format="value(projectNumber)"

# Confirm which KSA the workload runs as.
kubectl get pod "{pod_name}" -n "{namespace}" \
  -o jsonpath='{.spec.serviceAccountName}'

# Identify the node the Pod runs on, then the node pool that node belongs to
# (Step 2 checks the node pool's Workload Identity mode).
NODE=$(kubectl get pod "{pod_name}" -n "{namespace}" -o jsonpath='{.spec.nodeName}')
kubectl get node "$NODE" \
  -o jsonpath='{.metadata.labels.cloud\.google\.com/gke-nodepool}'
```

--------------------------------------------------------------------------------

### Step 1: Capture the exact error signature

Read the workload's own logs and the `gke-metadata-server` logs to classify the
failure.

```bash
kubectl logs "{pod_name}" -n "{namespace}" --all-containers --prefix
kubectl describe pod "{pod_name}" -n "{namespace}"
```

Equivalent via Cloud Logging (preferred for historical events). Open it as a
**Logs Explorer deep link** — URL-encode the query and append the project and
time window:
`https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}`
(encode `/` as `%2F`, or use `;duration=PT1H` for a rolling hour):

```
resource.type="k8s_container"
resource.labels.namespace_name="{namespace}"
resource.labels.pod_name="{pod_name}"
severity>=WARNING
```

**Classify the signature:**

-   `iam.serviceAccounts.getAccessToken` denied / `HTTP/403` → the workload is
    using the **GSA impersonation** path. This is expected for the legacy setup,
    but if you intend to use **direct binding**, it means the KSA still carries
    a leftover `iam.gke.io/gcp-service-account` annotation (or the client SDK is
    configured to impersonate) and is unintentionally impersonating a GSA; go to
    Step 3.
-   `403 PERMISSION_DENIED` on the **target API/resource** (no `getAccessToken`
    in the error) → the resolved identity lacks the required IAM role on that
    resource; go to Step 4.
-   `could not find default credentials` / `cannot find the metadata server` →
    metadata-server connectivity or a startup race; go to Step 5.
-   Calls succeed but as the **node default service account** → Workload
    Identity is not in effect for this node pool; go to Step 2.

--------------------------------------------------------------------------------

### Step 2: Verify Workload Identity is enabled (cluster + node pool)

Both the cluster and the **node pool the Pod runs on** must have Workload
Identity enabled. A node pool with `GCE_METADATA` (instead of `GKE_METADATA`)

From google/skills, Apache-2.0.

More from google/skills