observability-sre-triage

Files (4)

• SKILL.md
• apm-signals.md · references/
• log-investigation.md · references/
• slo-and-alerts.md · references/

SKILL.md (first lines)

# SRE Service Triage

Decide whether a service is healthy, degraded, or unhealthy, and say why. Triage is a hierarchy, not a checklist: SLOs
and alerts define whether the service is failing its contract, trace-derived golden signals describe how it is failing,
dependencies and infrastructure explain where the failure comes from, and logs supply the sentence you put in the
incident channel. Work down the hierarchy until the evidence supports a verdict, then stop.

For authoring and tuning SLO definitions, burn-rate rules, and alert thresholds, use the
**observability-service-reliability** skill. This skill only reads that state. For Kubernetes workload, node, or
control-plane diagnosis — restart loops, OOM kill confirmation, node pressure, admission rejections, stuck rollouts —
hand off to the **observability-k8s-investigation** skill. This skill checks whether a Kubernetes-hosted service is
saturated; it does not diagnose why the pod or the node behind it is failing.

<!-- begin-partial: preamble -->

## Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the
[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
directly.

<!-- end-partial: preamble -->

### Analysis without cluster access

The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the
evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason
from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific
query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is
useful without a cluster; one that only asks for setup is not.

Everything here is expressed in ES|QL (`POST /_query`) or the Kibana Observability APIs. Do not use Query DSL, and do
not use the ES|QL `KQL` search function — express predicates natively (`WHERE service.name == "checkout"`).

## Jobs to be done

- Answer "is service X healthy?" with a verdict and the evidence behind it
- Answer "why is service X slow / erroring / quiet?" by localizing the change to the service, a dependency, or its
  infrastructure
- Read SLO status, burn rate, and remaining error budget during an incident
- Determine which alerting rules currently apply to a service, including all-services rules
- Funnel a noisy log stream down to the failures that explain the degradation
- Identify which attribute (version, host, pod, region, route) distinguishes the failing or slow subpopulation
- Distinguish a healthy service from a service with no telemetry

## Output discipline

Applies to every response produced under this skill.

- **Commit to the best-supported conclusion.** When the evidence points one way, say so. Do not downgrade confidence to
  sound cautious — hedging on unambiguous evidence is a defect, not humility.
- **Commit to a verdict**: healthy, degraded, or unhealthy, followed by the reason. A triage answer that does not name
  one of the three has not done the job.
- **State confidence once**, in the conclusion. Do not restate it per bullet.
- **Do not speculate past the evidence.** If the telemetry did not show a cause, it does not go in the answer. Name what
  is unknown and stop. Never offer a mechanism ("probably a GC pause", "likely a noisy neighbor") that no signal
  measured.
- **Report absence as absence.** Zero rows means the data is missing or not collected; it never means the underlying
  condition is healthy. "No dependency metrics" is not "dependencies are fine".
- **Do not pad.** No restating the question, no narrating which queries were run unless the result mattered, no
  summarizing the summary.
- **End on the finding.** No trailing offers such as "want me to dig deeper?". Actionable follow-ups belong in a
  recommendations list, phrased as recommendations, not as questions.

## Signal hierarchy

Signals disagree constantly. This ordering decides which one wins.

| Rank | Signal                                | Authority                                                                                   |
| ---- | ------------------------------------- | ------------------------------------------------------------------------------------------- |
| 1    | **SLO status and burn rate**          | Authoritative when SLOs exist. They encode the agreed definition of "good" for this service |
| 2    | **Active alerting rules**             | Authoritative when no SLO covers the symptom. Sourced from the Alerting API                 |
| 3    | **Error rate, latency, throughput**   | Describes the degradation. Decisive only when nothing above it exists                       |
| 4    | **Dependency health**                 | Locates the cause upstream or downstream; does not by itself set the verdict                |
| 5    | **ML anomalies**                      | Deviation from learned baseline, not from a target. Corroborates and time-bounds            |
| 6    | **Infrastructure (CPU, memory, OOM)** | Explains a mechanism. A saturated pod with healthy golden signals is a risk, not an outage  |
| 7    | **Logs**                              | Explain, never decide. Log volume is not health                                             |

Conflict rules:

- **SLO healthy, latency elevated** → degraded but within error budget. The verdict follows the SLO; report the trend as
  a risk with the burn rate.
- **SLO violated, current-window metrics look fine** → trust the SLO and check its window. SLOs are evaluated over hours
  or days; a 15-minute ES|QL window can look clean while the budget is already spent.
- **Alerts firing, no SLO defined** → the alerts are the verdict. Resolve each rule's `params` to confirm it actually
  targets this service before attributing it.
- **Logs noisy, golden signals flat** → not degraded. High log volume without an error-rate or latency change is a
  logging-configuration finding, not a health finding.
- **Throughput collapsed, error rate flat** → the caller stopped calling. Look upstream before blaming this service.
- **Any query returns zero rows** → missing data. Say which signal is unavailable and lower the scope of the verdict
  accordingly; never convert silence into health.

## Routing: symptom to first signal

| Presenting symptom                      | Pull first                                                           | Reference                                               |
| --------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------- |
| "Is X healthy?" / unclear               | SLO status, then active rules, then golden signals                   | [slo-and-alerts.md](references/slo-and-alerts.md)       |
| "X is slow"                             | Latency percentiles versus the prior period, then dependency latency | [apm-signals.md](references/apm-signals.md)             |
| "X is erroring" / 5xx                   | Error rate by route, then failed-transaction correlation             | [apm-signals.md](references/apm-signals.md)             |
| "X is down" / no traffic                | Throughput, then confirm the service still ingests at all            | [apm-signals.md](references/apm-signals.md)             |
| "Only some requests are bad"            | Subpopulation correlation over candidate attributes                  | [apm-signals.md](references/apm-signals.md)             |
| "An alert fired" / "the SLO is burning" | Rule `params` and SLO burn rate, then the metric the rule watches    | [slo-and-alerts.md](references/slo-and-alerts.md)       |
| "What is in the logs?" / noisy logs     | The log funnel — iterate with `NOT` exclusions                       | [log-investigation.md](references/log-investigation.md) |
| Suspected OOM, throttling, restarts     | Container CPU and memory limit utilization                           | [apm-signals.md](references/apm-signals.md)             |
| "Is it saturated?" on a non-K8s host    | Host CPU, memory, and load average from the hostmetrics receiver     | [apm-signals.md](references/apm-signals.md)             |
| "Which downstream is hurting X?"        | Per-destination call volume, latency, and failure rate               | [apm-signals.md](references/apm-signals.md)             |

## Data sources

OTel-native data streams, verified against Elasticsearch 9.6.0:

| Data                          | Index pattern                                                                                               |

From elastic/agent-skills, Apache-2.0.

More from elastic/agent-skills