observability-llm-obs

Files (4)

• SKILL.md
• esql-recipes.md · references/
• genai-fields.md · references/
• integrations.md · references/

SKILL.md (first lines)

# LLM and Agentic Observability

Answer questions about monitoring LLMs and agentic components using **data actually ingested into Elastic** — nothing
else. The four questions this skill answers are LLM performance, cost and token utilization, response quality, and call
chaining or agentic workflow orchestration.

A given deployment typically uses **one or more** ingestion paths: APM/OTLP traces, and/or integration metrics and logs.
Which one exists is a discovery result, not an assumption — never assume both are present. For ES|QL syntax, commands,
and query patterns, use the **elasticsearch-esql** skill. For service-level latency and error triage that is not
LLM-specific, use the **observability-sre-triage** skill.

<!-- begin-partial: preamble -->

## Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the
[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
directly.

<!-- end-partial: preamble -->

### Analysis without cluster access

The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the
evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason
from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific
query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is
useful without a cluster; one that only asks for setup is not.

## Jobs to be done

- Discover which LLM ingestion path a deployment actually uses before querying anything
- Discover the real LLM field names in this deployment, because GenAI attribute naming is not consistent
- Report LLM latency, throughput, and error rate by model, service, or time
- Report token utilization, and report cost only when a cost field genuinely exists
- Investigate response quality: failures, timeouts, finish reasons, content filters, and guardrail events
- Reconstruct agentic call chains and find the bottleneck span in a multi-step workflow
- Correlate findings with SLOs and alerting rules defined on LLM-related data

## Output discipline

Applies to every response produced under this skill.

- **Commit to the best-supported conclusion.** When the evidence points one way, say so. Do not downgrade confidence to
  sound cautious — hedging on unambiguous evidence is a defect, not humility.
- **State confidence once**, in the conclusion. Do not restate it per bullet.
- **Do not speculate past the evidence.** If a cause was not observed, it does not go in the answer. Say what is unknown
  and stop.
- **Report absence as absence.** Zero rows means the data is missing or not collected; it never means the underlying
  condition is healthy. In particular, **if no cost field exists, say cost is not instrumented** — do not multiply token
  counts by a guessed per-token price and present the product as a cost figure.
- **Report the numbers actually found.** Quote the token counts, latencies, and error rates the query returned. Do not
  round them into vague characterizations, and do not carry forward a number from an example in this skill.
- **Do not pad.** No restating the question, no narrating which queries were run unless the result mattered, no
  summarizing the summary.
- **End on the finding.** No trailing offers such as "want me to dig deeper?". Actionable follow-ups belong in a
  recommendations list, phrased as recommendations, not as questions.

## Where the data lives

| Ingestion path                  | Index patterns                                                            | What it can answer                                                       |
| ------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| OTel / EDOT traces via OTLP     | `traces-*.otel-*`, or the generic `traces-*`                              | Per-request latency, tokens, models, finish reasons, **and call chains** |
| Elastic APM agent traces        | `traces-apm*`, or the generic `traces-*`                                  | Per-request latency and outcome; GenAI attributes if the SDK adds them   |
| APM/OTel metrics                | `metrics-apm*`, `metrics-*.otel-*`                                        | Aggregated service-level rates and latencies                             |
| Elastic LLM integration metrics | `metrics-<integration>.*` (for example `metrics-aws_bedrock_agentcore.*`) | Aggregate token counts, invocations, latency, sometimes cost             |
| Elastic LLM integration logs    | `logs-<integration>.*`                                                    | Prompt/response records, guardrail and content-filter events             |

Use the generic `traces-*` pattern to find trace data regardless of whether an Elastic APM agent or OpenTelemetry
collected it. Instrumentation can come from EDOT, OpenLLMetry, OpenLIT, or Langtrace exporting to OTLP — all of them
land LLM and agent spans in the trace data streams.

**Only traces can reconstruct a call chain.** `trace.id`, `span.id`, and `parent.id` are the only way to rebuild an
agentic call chain. Integration metrics are pre-aggregated and cannot do it — if the question is about orchestration and
only integration metrics exist, say the chain cannot be reconstructed from the available data.

Background reading:
[LLM and agentic AI observability](https://www.elastic.co/docs/solutions/observability/applications/llm-observability),
[EDOT LLM use cases](https://www.elastic.co/docs/solutions/observability/get-started/opentelemetry/use-cases/llms),
[Observability Labs — LLM Observability](https://www.elastic.co/observability-labs/blog/tag/llmobs).

## Process: determine what data is available

Run these in order. Do not skip to step 5.

1. **Verify the connection and detect the version.** Call `GET /`. The decision this drives is which query language
   surface is available: `build_flavor: "serverless"` means all ES|QL features including `TS`, `TRANGE`, and `TBUCKET`
   are available. Otherwise use `version.number`: on Stack, `TS` is preview in 9.2 and GA in 9.4, and `TRANGE` is GA in
   9.3.0, so treat **Stack 9.4** as the floor for the whole time-series path and fall back to `FROM` with
   `BUCKET(@timestamp, ...)` below it. Getting this wrong produces a query that fails to parse.

2. **Determine which ingestion paths exist.** The decision is whether this deployment has traces, integration data, or
   both — and it changes which questions are answerable at all. List data streams with `GET /_data_stream/<name>` (pass
   `traces-*`, `metrics-*`, `logs-*`) or resolve patterns with `GET /_resolve/index/<pattern>`. Look for trace data
   streams, `metrics-apm*`, and any `metrics-*` or `logs-*` matching a known LLM integration dataset. If neither path
   has LLM data, say so and stop — do not answer from general knowledge about the model provider.

3. **Discover the real LLM field names.** The decision is which exact field paths to write into ES|QL, and it cannot be
   guessed. Naming differs across instrumentations: `gen_ai.*` versus `llm.*` versus integration-specific names, and the
   semantic conventions themselves have shifted (older instrumentation emits `gen_ai.system`, newer emits
   `gen_ai.provider.name`). Use `GET /_field_caps` with a field pattern such as `*gen_ai*`, `*llm*`, `*token*`,
   `*cost*`, or read `GET /<index>/_mapping`, then sample a document to confirm the values are populated. **Attribute
   nesting differs by ingestion path** — OTel-native trace data streams expose span attributes as `attributes.<name>`
   (and often as a bare passthrough `<name>`), not as `span.attributes.<name>`. Confirm which form resolves before
   writing a query. See [references/genai-fields.md](references/genai-fields.md) for the attribute catalog and the
   resolution rules.

4. **Decide whether a cost field exists.** Cost is **not** part of the OpenTelemetry GenAI specification. Some
   instrumentations add a custom attribute such as `llm.response.cost.usd_estimate`, and some integrations expose a cost
   metric, but many deployments have neither. Look for it explicitly in step 3. If it is absent, the answer to a cost
   question is that cost is not instrumented — report token counts instead and name the gap.

From elastic/agent-skills, Apache-2.0.

More from elastic/agent-skills