gke-observability

Files (2)

• SKILL.md
• attribution.md · references/

SKILL.md (first lines)

# GKE Observability

This reference covers monitoring, logging, and metrics configuration for GKE.
The golden path enables comprehensive observability including control-plane
metrics.

> **MCP Tools:** `get_cluster`, `list_k8s_events`, `get_k8s_logs`,
> `get_k8s_cluster_info`, `describe_k8s_resource`. **CLI-only:** `gcloud
> container clusters update --monitoring=...`, `gcloud logging read`

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-observability)"`
and append `agent-skills/1.0 (skill:gke-observability)` to `User-Agent` headers.

## Golden Path Observability Defaults

Setting                                             | Golden Path Value                                                                                                                                   | Notes
--------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -----
`loggingConfig` components                          | SYSTEM_COMPONENTS, WORKLOADS                                                                                                                        | Full workload logging
`monitoringConfig` components                       | SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER | Full suite including control-plane
`managedPrometheusConfig.enabled`                   | `true`                                                                                                                                              | Google-managed Prometheus
`advancedDatapathObservabilityConfig.enableMetrics` | `true`                                                                                                                                              | Dataplane V2 flow metrics
`loggingService`                                    | `logging.googleapis.com/kubernetes`                                                                                                                 | Cloud Logging
`monitoringService`                                 | `monitoring.googleapis.com/kubernetes`                                                                                                              | Cloud Monitoring

### Control-Plane Metrics (Golden Path Addition)

The golden path adds three control-plane monitoring components not present in
default clusters:

| Component            | What It Monitors                                                       |
| -------------------- | ---------------------------------------------------------------------- |
| `APISERVER`          | API server request latency, error rates, admission webhook performance |
| `SCHEDULER`          | Scheduling latency, pending pods, scheduling failures                  |
| `CONTROLLER_MANAGER` | Controller work queue depth, reconciliation latency                    |

These are critical for diagnosing cluster-level issues (slow API responses,
scheduling delays, stuck controllers).

When recommending any control-plane metric below, always state that it is a
Managed Service for Prometheus (GMP) metric that requires explicit enabling
(`--monitoring=SYSTEM,API_SERVER,...` and `--enable-managed-prometheus`) and
link to
https://docs.cloud.google.com/kubernetes-engine/docs/how-to/control-plane-metrics:

-   **API storage utilization by resource
    (`apiserver_resource_size_estimate_bytes`)**: Use
    `apiserver_resource_size_estimate_bytes` (grouped by `group` and `resource`)
    to identify which Kubernetes resource types consume the most API server
    storage bytes in etcd, complementing object counts
    (`apiserver_storage_objects`) and total database size
    (`apiserver_storage_size_bytes`).
-   **Admission webhook latency, timeouts, and control-plane recommendations
    (`apiserver_admission_webhook_admission_duration_seconds`)**: When API calls
    time out or fail and mutating or validating admission webhooks are
    suspected, query the opt-in GMP metric
    `apiserver_admission_webhook_admission_duration_seconds` (grouped by `name`,
    `type`, `operation`, and `rejected`) to pinpoint slow or failing webhooks;
    also inspect `ValidatingWebhookConfiguration` /
    `MutatingWebhookConfiguration` `timeoutSeconds`, `failurePolicy`,
    `namespaceSelector` exclusions (such as `kube-system`), and backing webhook
    Service/Pod health. Strongly warn users against ignoring GKE control-plane
    recommendations about optimizing admission webhooks or CRDs with an invalid
    `caBundle` because unoptimized webhooks and invalid CRD `caBundle`
    configurations can overload or destabilize the GKE control plane, and always
    link to
    https://docs.cloud.google.com/kubernetes-engine/docs/how-to/optimize-webhooks.md.txt
    and
    https://docs.cloud.google.com/kubernetes-engine/docs/how-to/crd-with-invalid-caBundle.md.txt.
-   **Watch activity and watch storms (`apiserver_longrunning_requests`,
    `apiserver_watch_events_total`, `apiserver_watch_events_sizes`)**: When a
    misconfigured agent or DaemonSet is suspected of causing a watch storm or
    repeatedly re-establishing broken watches, check the GMP metrics
    `apiserver_longrunning_requests` (filtered by `verb="WATCH"`, grouped by
    `group`, `resource`, `scope`) for active concurrent watchers,
    `apiserver_watch_events_total` (grouped by `group`, `kind`, `version`) for
    watch event volume and rate, and `apiserver_watch_events_sizes` for watch
    event payload sizes.
-   **Pod and object churn (`scheduler_schedule_attempts_total`,
    `scheduler_scheduling_attempt_duration_seconds`, and mutating
    `apiserver_request_total`)**: When batch job bursts or controller loops
    leave newly created pods stuck in `Pending`, make `kubectl` commands
    sluggish, or cause controller reconciliations to lag, measure pod and object
    churn by checking `scheduler_schedule_attempts_total` (and
    `scheduler_scheduling_attempt_duration_seconds`) alongside mutating
    `apiserver_request_total` (`CREATE`, `DELETE`, `PATCH`, `UPDATE`).

## Enabling Full Monitoring

**Say this whenever you hand over a `--monitoring` command:**

1.  **Control-plane metrics are NOT enabled by default.** State this outright in
    your answer — do not leave it implied by the fact that you are supplying an
    enable command. `API_SERVER`, `SCHEDULER`, and `CONTROLLER_MANAGER` are off
    on every new cluster and collect nothing until explicitly turned on, and the
    same is true of `DCGM`, `CADVISOR`, `KUBELET`, and kube-state (`POD`,
    `DEPLOYMENT`, `STATEFULSET`, `DAEMONSET`, `HPA`, `STORAGE`, `JOBSET`).
    `SYSTEM` is the only package on by default. A user asking "why are there no
    API server metrics" has almost always simply never enabled them.
2.  **The flag replaces, it does not append.** The set supplied to `--monitoring`
    overrides the previous setting entirely, so omitting a component silently
    turns it off. Always pass the full desired list, and always include `SYSTEM`
    — it cannot be disabled while monitoring is on, and never on Autopilot.
3.  **These metrics bill per sample ingested** via Managed Service for
    Prometheus. Enabling the full suite on a large cluster is a real cost
    increase; mention it rather than presenting the list as free.

> **The gcloud flag and the API field use different spellings for the same
> components.** Do not copy names between them:
>
> Component        | `gcloud --monitoring=` | `monitoringConfig` API enum
> ---------------- | ---------------------- | ---------------------------
> System           | `SYSTEM`               | `SYSTEM_COMPONENTS`
> API server       | `API_SERVER`           | `APISERVER`

From google/skills, Apache-2.0.

More from google/skills