observability-service-reliability

Files (4)

• SKILL.md
• burn-rate-rules.md · references/
• documentation.md · references/
• slo-api-schemas.md · references/

SKILL.md (first lines)

# Service Reliability

Design reliability targets that people will actually act on, then operate them. This skill covers the judgment before
the API call — which service-level indicator fits the data you have, what target is achievable rather than aspirational,
whether an SLO is even the right instrument — and then the mechanics of creating, alerting on, resetting, and retiring
SLOs through the Kibana API.

Reliability instruments are not interchangeable. An SLO measures a user-visible outcome against a spendable budget; a
threshold rule fires on a raw condition; an anomaly job finds deviations where no fixed threshold exists; a synthetics
monitor is the only one of the four that can see a service that has stopped emitting telemetry entirely. Choosing wrong
produces alerts that are technically correct and operationally useless. For diagnosing a service that is already
degraded, and for the incident workflow itself, use the **observability-sre-triage** skill; for general rule lifecycle
mechanics use the **kibana-alerting-rules** skill.

<!-- begin-partial: preamble -->

## Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the
[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
directly.

<!-- end-partial: preamble -->

### Analysis without cluster access

The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the
evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason
from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific
query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is
useful without a cluster; one that only asks for setup is not.

SLO and alerting operations run against Kibana and use the `kbn:` prefix (for example,
`POST kbn:/api/observability/slos`); data validation runs against Elasticsearch with a bare path (for example,
`POST /_query`). Kibana SLO commands are space-scoped — pass the space explicitly. For non-default spaces the HTTP path
becomes `kbn:/s/<space_id>/api/observability/slos`.

Full request-body schemas for every SLI type live in [references/slo-api-schemas.md](references/slo-api-schemas.md), the
burn-rate rule schema in [references/burn-rate-rules.md](references/burn-rate-rules.md), and curated official
documentation in [references/documentation.md](references/documentation.md).

## Jobs to be done

- Translate a reliability concern into the right instrument: SLO, burn-rate rule, threshold rule, anomaly job, or
  synthetics monitor
- Choose an SLI type for a given service and data shape, and validate the underlying fields before committing
- Set a target, time window, and budgeting method that are defensible against measured history
- Decide whether to group an SLO, and refuse high-cardinality grouping
- Create, verify, update, reset, disable, and delete SLOs
- Attach burn-rate alert rules with fast-burn and slow-burn windows routed to different severities
- Keep the alerting surface signal-dense: remove duplicate coverage, snooze instead of disable, delete unactionable
  rules
- Monitor availability of user-facing endpoints with synthetics and feed that into an availability SLI

## Output discipline

Applies to every response produced under this skill.

- **Commit to the best-supported conclusion.** Recommend one SLI design and defend it. Do not enumerate every SLI type
  with equal weight and leave the choice to the user — that is not a recommendation, it is a menu.
- **State the target and window you chose, and why, once.** Do not restate the justification per bullet.
- **Do not invent field names.** Verify every field against the mapping before writing it into an indicator. If a field
  cannot be confirmed, say so and stop; a plausible-looking field name is worse than an admitted gap.
- **Do not speculate past the evidence.** If the measured history does not support a target, say what it does support.
- **Report absence as absence.** Zero events means the data is missing or the service is not emitting; it never means
  the service is healthy. An SLI ratio computed over zero total events is undefined, not 100%.
- **Do not pad.** No restating the question, no narrating which queries were run unless the result mattered.
- **End on the finding.** No trailing offers such as "want me to set this up?". Actionable follow-ups belong in a
  recommendations list, phrased as recommendations, not as questions.

## Process: route the reliability concern to an instrument

Do this before designing anything. Most bad SLOs are threshold rules wearing a costume.

1. **Name the user-visible symptom.** Ask what a user or downstream consumer would notice and complain about. If the
   answer is a resource number rather than an experience — disk at 90%, heap climbing, replica lag — the concern is a
   capacity ceiling, not a reliability target. Route it to a threshold rule.

2. **Check whether coverage already exists.** List SLOs with `GET kbn:/api/observability/slos`, alerting rules with
   `GET kbn:/api/alerting/rules/_find`, and anomaly jobs with `GET /_ml/anomaly_detectors`. Adding a second instrument
   on a signal that is already covered is the most common source of duplicate pages.

3. **Pick the instrument.**

   | Concern                                                                      | Instrument               | Why, and what it costs                                                                                                                                                      |
   | ---------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | A user-visible success or latency outcome you want to budget over weeks      | SLO + burn-rate rule     | Gives a spendable budget and history, and paces change velocity. Costs a transform, needs steady traffic, and is deliberately slow — even fast-burn uses a one-hour window. |
   | A hard bound with a known safe value and an immediate operator action        | Threshold or custom rule | Fires within one schedule interval, no transform. But it has no budget and no memory, so it re-fires for as long as the condition holds.                                    |
   | A signal with no fixed threshold, strong seasonality, or many entities       | Anomaly detection job    | Finds unknown-unknowns and adapts to seasonality. Needs weeks of history to be trustworthy and emits scores, not outcomes, so it is a poor pager.                           |
   | Reachability of a user-facing endpoint from outside your own telemetry       | Synthetics monitor       | The only instrument that detects a total outage. Costs a check budget (see the synthetics section) and cannot explain an internal partial failure.                          |
   | An internal job with no consumer contract, or a service still changing daily | None                     | An SLO with no owner and no stable baseline becomes permanently red and is then ignored, which is worse than no SLO.                                                        |

4. **Do not skip the outage case.** An SLI built from a service's own logs or traces cannot see the service disappear:
   with zero events there is no `total`, so the ratio is undefined and the SLO neither burns nor recovers. Pair every
   request-based SLO on a user-facing service with either a synthetics availability monitor or a no-data threshold rule.
   This is the one place where two instruments on one signal is correct rather than duplicative.

## Process: design the SLO

1. **Locate the data and confirm the fields exist.** Resolve the index pattern with `GET /_resolve/index/<pattern>`,
   then confirm every field the indicator will reference with `GET /<index>/_mapping` or `GET /_field_caps`. OTel-native
   data lives in `traces-*.otel-*`, `metrics-*.otel-*`, and `logs-*.otel-*`, with `service.name` populated on all three;
   APM aggregate metrics live in `metrics-service_summary.1m.otel-*`, `metrics-service_transaction.1m.otel-*`,
   `metrics-transaction.1m.otel-*`, and `metrics-service_destination.1m.otel-*`. Do not assume a status-code, duration,
   or outcome field exists because it is conventional — confirm it. A wrong field name produces an SLO that computes
   cleanly and means nothing; [How a missing field fails](#how-a-missing-field-fails) shows exactly how.

   Two field placements are worth knowing because they are commonly guessed wrong: `http.response.status_code` is on
   `traces-*.otel-*` and not on `logs-*.otel-*`, and `transaction.duration.us` is on `traces-*.otel-*` and not on the
   `metrics-*.1m.otel-*` rollups, which carry `transaction.duration.histogram` and `transaction.duration.summary`
   instead. Confirm both against the deployment in front of you rather than trusting this list.

From elastic/agent-skills, Apache-2.0.

More from elastic/agent-skills