observability-k8s-investigation

Files (3)

• SKILL.md
• failure-modes.md · references/
• query-recipes.md · references/

SKILL.md (first lines)

# Kubernetes Investigation

Diagnose Kubernetes issues using OTel telemetry collected via EDOT (Elastic Distribution of OpenTelemetry) and the
kube-stack collector. Correlate cluster state, pod runtime metrics, K8s events, application logs, and APM to identify
root cause across the workload, node, and control-plane layers.

<!-- begin-partial: preamble -->

## Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the
[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
directly.

<!-- end-partial: preamble -->

### Analysis without cluster access

The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the
evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason
from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific
query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is
useful without a cluster; one that only asks for setup is not.

Every ES|QL query in this skill and in [references/query-recipes.md](references/query-recipes.md) runs via
`POST /_query`. Alert state is read with `GET kbn:/api/alerting/rules/_find`. Field-presence checks use
`GET /<index>/_mapping` or `GET /_field_caps`. The [Operations](#operations) table maps each to its `elastic` CLI
equivalent.

## Scope

**In scope:** OTel-receiver-namespaced indices (`metrics-kubeletstatsreceiver.otel-*`,
`metrics-k8sclusterreceiver.otel-*`, `logs-k8seventsreceiver.otel-*`, `logs-k8sobjectsreceiver.otel-*`) and OTel
semantic conventions (`k8s.pod.name`, `k8s.namespace.name`, `k8s.container.restarts`).

**Out of scope:**

- The legacy Elastic Agent Kubernetes integration (`metrics-kubernetes.*`, `logs-kubernetes.*`, `kubernetes.*` fields).
  Being deprecated — do not author queries against these paths.
- APM-layer analysis (service SLO breaches, transaction error rates, upstream dependency health). Different domain —
  once a K8s root cause is ruled in or out, hand off to the **observability-sre-triage** skill, which owns SLO status
  and burn rate, active alerting rules, throughput, latency, error rate, dependency health, and log funnelling. That is
  also the skill to use when the workload turns out not to be Kubernetes-hosted at all.
- Cluster provisioning, capacity planning, cost optimization. Different domain.

## Guidelines

These apply to every investigation. When in doubt, re-read them before writing the synthesis.

**Absence of evidence is not evidence. Do not confabulate from empty results.** If log queries return 0 rows, logs are
likely not collected or the pod has no recent lines — this does _not_ mean "dependency unavailable" or any other
specific failure mode. Report `no_logs_available` and weight remaining signals accordingly.

**Empty dependency data ≠ upstream healthy.** Services without APM instrumentation (load generators, workers) emit no
destination metrics. Report `insufficient_dependency_data`, not "upstreams OK."

**Co-symptoms are not causes.** Two services degrading simultaneously usually share an upstream, not a causal link. Only
attribute causation when (a) one service's degradation clearly precedes the other's, and (b) the delta is large (>5×
error rate, >3× latency).

**OOMKilled ≠ memory leak by default.** The limit might simply be undersized for the workload's working set. Tell them
apart by the shape of the memory curve: a monotonic climb to the limit that resets on each restart, with load flat
against the prior week and no recent deploy, is the leak signature — commit to it at high confidence. Reach for a 7-day
same-hour baseline when the shape is ambiguous — spiky, diurnal, or load-correlated — not as a precondition for every
OOMKilled finding.

**Error-termination ≠ application bug by default.** Check `k8s.container.cpu_limit_utilization` first. CFS throttling
driving liveness probe timeouts is the most common misdiagnosis in this space.

**Average CPU hides throttling.** A pod can look healthy at 40–60% average `cpu_limit_utilization` while being throttled
severely at p99. Linux enforces CPU limits in 100ms periods; bursty workloads reach quota mid-period and stall. Look at
max and p95, not only the average.

**Restart count is boolean, not a counter.** `k8s.container.restarts` is pulled directly from the K8s API and can be
pruned by the kubelet at any time, so the absolute value is unreliable. Treat it as `== 0` (no recent restarts) versus
`> 0` (recently restarting); do not derive backoff timing or "linear versus exponential" patterns from it. Confirm the
restart pattern via K8s `Killing` / `BackOff` events instead.

**Prefer to report uncertainty over manufacturing confidence.** If the evidence is ambiguous, the synthesis should say
so. Competing hypotheses are a valid output.

**Equally, do not manufacture uncertainty.** The rule above is about ambiguous evidence, not about tone. When the
pivotal signal is present and corroborated, commit to it at high confidence. Hedging an unambiguous finding down to
"medium" is as much a defect as overclaiming.

**Deliver the synthesis and stop.** State confidence once, in the HYPOTHESIS line — not again per bullet. Do not narrate
which queries were run unless a result changed the conclusion, and do not restate the alert back to the reader. End on
RECOMMENDED NEXT STEPS or DOWNSTREAM IMPACT; never close with an offer such as "want me to look further?". Follow-up
work belongs in the recommendations list, phrased as a recommendation.

## Indices and fields

### Where to look

| Signal                | Index pattern                                       | Use                                                                 |
| --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- |
| Pod/container runtime | `metrics-kubeletstatsreceiver.otel-*`               | CPU, memory, network, filesystem. Utilization ratios.               |
| Cluster state         | `metrics-k8sclusterreceiver.otel-*`                 | Restarts, phase, last-terminated reason, HPA, quota, node condition |
| K8s events            | `logs-k8seventsreceiver.otel-*`                     | Killing, BackOff, FailedScheduling, Evicted, image pull events      |
| K8s object snapshots  | `logs-k8sobjectsreceiver.otel-*`                    | Deployment/service/configmap state over time                        |
| Application logs      | `logs-*.otel-*`                                     | `body.text`, `severity_text`, filtered by `k8s.pod.name`            |
| APM                   | `traces-*.otel-*`, `metrics-service_*.otel-default` | Correlate via `service.name` + K8s resource attrs                   |
| ML anomalies          | `.ml-anomalies-*`                                   | Memory-growth, restart-rate, throttle jobs (if configured)          |

### Key fields

Flat OTel paths work in ES|QL. Prefer the flat form for readability; the nested `resource.attributes.*` form is for raw
log documents only.

| Field                                            | Index                       | What it is                                              |
| ------------------------------------------------ | --------------------------- | ------------------------------------------------------- |
| `k8s.pod.name`                                   | all k8s                     | Pod name                                                |

From elastic/agent-skills, Apache-2.0.

More from elastic/agent-skills