gke-alert-configuration

Files (6)

• SKILL.md
• attribution.md · references/
• gke_configuration_prerequisites.md · references/
• metrics_and_alerts_catalog.md · references/
• promql_queries.md · references/
• validate_config.py · scripts/

SKILL.md (first lines)

# GKE Alert Configuration

This skill provides guidelines and best practices for creating robust,
high-signal alerting policies for Google Kubernetes Engine workloads using
Google Cloud Managed Service for Prometheus and Terraform. It ensures
comprehensive coverage of the **4 Golden Signals** and key cluster health
metrics while minimizing alert noise.

--------------------------------------------------------------------------------

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-alert-configuration)"`
and append `agent-skills/1.0 (skill:gke-alert-configuration)` to `User-Agent` headers.

## Critical Rules

*   **Negative Triggers and Scope Redirection for Non-GKE Standalone Runtimes**:
    *   This skill is strictly scoped to Google Kubernetes Engine (GKE)
        workloads, clusters, and services using PromQL and Google Cloud Managed
        Service for Prometheus.
    *   **Do not use for non-GKE compute runtimes**, such as standalone Compute
        Engine virtual machines or standalone Cloud Run services without GKE.
    *   **STOP AND RESPOND DIRECTLY (Do Not Edit Files)**: When the user
        requests alert configuration for non-GKE compute infrastructure:
        1.  **Do not write, create, edit, or validate any Terraform files on
            disk**.
        2.  **Immediately stop and respond directly to the user in chat**:
            *   **Explicitly Clarify Out-of-Scope**: State clearly that
                standalone Compute Engine virtual machine monitoring or
                standalone Cloud Run monitoring is out of scope for this
                GKE-specific PromQL alerting skill, which is designed
                specifically for GKE workloads using Google Cloud Managed
                Service for Prometheus and PromQL.
            *   **Do Not Generate GKE PromQL Alerts**: Do not create or generate
                Kubernetes PromQL alert policies or fabricate Kubernetes
                container, pod, or node resources for non-GKE infrastructure.
            *   **Redirect the User**: Guide and redirect the user to standard
                Google Cloud Monitoring metrics, such as
                `compute.googleapis.com/instance/cpu/utilization` or
                `run.googleapis.com/request_latencies`, using standard
                `google_monitoring_alert_policy` with `condition_threshold` or
                MQL, or recommend the relevant specialized Cloud observability
                skill.
*   **Mandatory `kube-state-metrics` (KSM) Cost Guardrail**:
    *   Deploying open-source `kube-state-metrics` in Google Cloud Managed
        Service for Prometheus incurs billable metric ingestion costs.
    *   **STOP AND ASK PERMISSION FIRST (Do Not Edit Files)**: When a requested
        alert rule relies on **Tier 2 KSM metrics** (such as `kube_cronjob_*`,
        `kube_pod_status_phase`, `kube_persistentvolume_*`, `kube_deployment_*`,
        `kube_statefulset_*`, `kube_job_*`, or `kube_daemonset_*`), **do not
        write, create, edit, or validate any Terraform files or generate alert
        policies before obtaining user approval**.
    *   Instead, you **must immediately stop and respond directly to the user**
        to:
        1.  **Alert the user** that the requested alert requires
            `kube-state-metrics`.
        2.  **Explain the cost impact**: Detail that `kube-state-metrics` incurs
            billable sample ingestion costs in Google Cloud Managed Service for
            Prometheus.
        3.  **Ask for explicit permission**: Ask the user for explicit
            permission before assuming, enabling, or generating KSM-dependent
            alert configurations.
        4.  **Recommend filtering or allowlisting**: Suggest and recommend
            filtering or allowlisting only the specific required metrics, such
            as using a `PodMonitoring` resource with `metricRelabeling`
            (`action: keep`) or KSM `--metric-allowlist` to minimize ingestion
            costs. Provide a concrete allowlist example.
    *   **Always prefer Non-KSM Native Alternatives** (Tier 1 cAdvisor or native
        GKE metrics documented in
        [metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md))
        whenever possible, such as using `container_memory_working_set_bytes`
        and `container_spec_memory_limit_bytes` instead of
        `kube_pod_container_resource_limits`.
    *   **Explicit Tier and Cost Surcharge Identification in Response**: In
        every response where you generate or recommend an alerting policy, you
        **must explicitly state its classification tier and cost impact**:
        *   **Tier 1 native or standard metric** (GKE built-in metrics, cAdvisor
            `container_*`, kubelet volume stats, kubelet node conditions, and
            control-plane metrics; see
            [metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md)):
            State that it is a **Tier 1 native or standard metric with zero KSM
            cost surcharge**.
        *   **Tier 2 KSM metric**: State that it is a **Tier 2 KSM-dependent
            metric** and follow the permission and allowlisting guardrail above.
            *(Tip: Generally, metrics with the `kube_` prefix that represent
            resource state or metadata belong to Tier 2).*
*   **Plan-Validate-Execute Loop for Approved File Edits**: When modifying,
    adding, or merging approved Terraform files on disk in a workspace, follow
    the three-phase workflow:
    1.  **Plan**: Draft a structured change plan (`changes.json`) containing
        proposed policy resource names, PromQL expressions, grouping labels, and
        durations.
    2.  **Validate**: Run the pre-edit validation script (`python3
        scripts/validate_config.py --plan changes.json`) to verify PromQL
        grammar, lookback windows, duration rules, and ensure no duplicate
        signals exist.
    3.  **Execute**: After the plan passes validation, apply or merge changes
        in-place into the target Terraform configuration (`alerts.tf`).
    4.  *Note*: When answering questions or providing Terraform snippets
        directly in chat where no disk modification is requested, output the
        complete, valid Terraform HCL block in your response.
*   **Configure the 4 Golden Signals and Cluster Health**: Always ensure the
    target Kubernetes workload or service has the following alerting coverage:
    1.  **Latency** (P95 response time)
    2.  **Errors** (Multi-Window Multi-Burn-Rate SLO alerts, such as Fast Burn 1
        hour / 5 minutes with factor 14.4, Slow Burn 6 hours / 30 minutes with
        factor 6.0; do not use simple static ratios)
    3.  **Traffic** (Sudden drop or complete metric disappearance using
        `absent()` or `default 0` syntax, or overload spikes)
    4.  **Saturation (Memory Limit Utilization Only)**: When describing or
        configuring alert policies for a cluster or project, include ONLY
        **Memory Saturation** (`container_memory_working_set_bytes` /
        `container_spec_memory_limit_bytes`). Do **NOT** include CPU saturation
        alerts or list `container_cpu_usage_seconds_total` as an alert metric

From google/skills, Apache-2.0.

More from google/skills