kibana-anomaly-detection

Files (12)

• SKILL.md
• README.md · references/
• agent-builder-companion.md · references/
• anomaly-detection-functions.md · references/
• investigation-queries.md · references/
• job-creation-recipes.md · references/
• observability-anomaly-expert.md · references/
• investigation.md · references/protocols/
• score-reference.md · references/
• security-anomaly-expert.md · references/
• troubleshooting-reference.md · references/
• worked-example.md · references/

SKILL.md (first lines)

# Elastic ML Anomaly Detection

Expert process for ML anomaly detection: attribute incidents to entities, explain scores and model behavior, diagnose
job lifecycle failures, and manage jobs. Read anomaly **results** from `POST /.ml-anomalies-*/_search` (Serverless-safe)
and **job/datafeed state** from ML REST APIs. When the user embeds fixture evidence (influencer rows, job stats) in the
prompt, apply the judgment below directly — do not re-fetch fields already supplied.

<!-- begin-partial: preamble -->

## Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the
[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
directly.

<!-- end-partial: preamble -->

## Mode selector

| User intent                                                                   | Mode                                                                                                   |
| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| "What broke?" / RCA / cross-job / blast radius / influencers / log categories | **Investigate**                                                                                        |
| "Why score high/low?" / renormalization / model bounds / forecasts            | **Explain**                                                                                            |
| Missing docs / memory limit / datafeed stopped / lifecycle / calendars        | **Troubleshoot**                                                                                       |
| Create a job / configure a datafeed / start analysis / retrieve results       | **Manage**                                                                                             |
| Security framing (attack chains, MITRE, exfil)                                | Investigate + [references/security-anomaly-expert.md](references/security-anomaly-expert.md)           |
| Observability/SRE framing (degradation, capacity, deployment regression)      | Investigate + [references/observability-anomaly-expert.md](references/observability-anomaly-expert.md) |

When a question spans modes: **Investigate → Explain → Troubleshoot**. Finish one mode before blending logic.

> **Serverless note:** Legacy `/_ml/anomaly_detectors/{job_id}/results/*` endpoints return HTTP 410 in Serverless.
> Always query `.ml-anomalies-*` via `POST /.ml-anomalies-*/_search` with `result_type` filters.

## Score quick reference

- `record_score` bands: **>75** critical · **50–75** warning · **25–50** minor · **<25** informational
- `multi_bucket_impact ≥ 3` → sustained shift (not a transient spike)
- `initial_record_score >> record_score` → renormalization (model saw worse anomalies later)
- `actual << typical` with `count`/`low_count`/`low_mean` → absence/outage, not just a low value
- Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity

> Full score definitions, renormalization mechanics, and `anomaly_score_explanation` components:
> [references/score-reference.md](references/score-reference.md).

## Core concepts

Treat `.ml-anomalies-*` as layered result types via `result_type` in search queries:

| `result_type`         | Scope           | Key fields                                                                               |
| --------------------- | --------------- | ---------------------------------------------------------------------------------------- |
| `bucket`              | Time window     | `anomaly_score`, `initial_anomaly_score`, `timestamp`                                    |
| `record`              | Detector row    | `record_score`, `initial_record_score`, `actual`, `typical`, `anomaly_score_explanation` |
| `influencer`          | Entity × bucket | `influencer_field_name`, `influencer_field_value`, **`influencer_score`**                |
| `model_plot`          | Bounds          | `model_lower`, `model_upper`, `actual`                                                   |
| `category_definition` | Log patterns    | `category_id`, `terms`, `regex`, `examples`                                              |

Read scores this way:

- `anomaly_score` / `record_score` = **current normalized** values (move as the model sees new extremes).
- `initial_anomaly_score` / `initial_record_score` = **immutable snapshots** from detection time.
- **`influencer_score` ranks entity responsibility within a bucket** — the highest score is the primary suspect, not the
  bucket-level `anomaly_score` alone.
- Map entities via `partition_field_value` / `by_field_value` / `over_field_value`.
- Read `multi_bucket_impact` (-5 to +5) to separate single-bucket spikes from sustained trends.

---

## Mode: Investigate — RCA

**When:** "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains.

### Process

1. **Discover jobs.** Call `GET /_ml/anomaly_detectors` when the job ID is unknown. Call
   `GET /_ml/anomaly_detectors/{job_id}` and `GET /_ml/datafeeds/datafeed-{job_id}` to learn source indices, entity
   fields (`by_field_name`, `over_field_name`, `partition_field_name`), and `bucket_span`. The decision: identify the
   related job group — jobs sharing a datafeed index or entity field monitor the same system from different angles.

2. **Scope the incident window.** Call `POST /.ml-anomalies-*/_search` with `result_type: bucket`, a time range, and
   optional minimum `anomaly_score`. The decision: fix the incident start/end and count how many jobs co-fire in that
   window. Low scores across many jobs simultaneously often indicate a systemic root cause.

3. **Attribute to entities (critical for RCA).** For the anomalous bucket timestamp, call
   `POST /.ml-anomalies-*/_search` with `result_type: influencer`, the job ID(s), and the bucket time range. Sort by
   **`influencer_score` descending**. The decision: name the entity with the **highest `influencer_score`** as the
   likely cause — it ranks how unusual each entity is in that bucket. Do not restate only the bucket `anomaly_score`
   without attributing responsibility. Recommend drilling into that entity's records next.

4. **Cross-job confirmation.** Re-query influencers (or bucket records) across related job IDs for the same entity
   values and time window. Entities anomalous in **2+ jobs** are prime suspects (resource fault or systemic failure);
   single-job entities are often downstream victims. See
   [references/protocols/investigation.md](references/protocols/investigation.md).

5. **Drill into records.** Call `POST /.ml-anomalies-*/_search` with `result_type: record`, exact job ID, entity filters
   (`partition_field_value`, `by_field_value`), and low minimum `record_score` (25 or lower). Read
   `multi_bucket_impact ≥ 3` as sustained behavioral shift. Read `actual` vs `typical` for fault class (spike vs
   absence/outage).

6. **Confirm with source evidence.** Call `POST /{index}/_search` on the datafeed source index for the suspect entity
   and time window. Raw source documents are ground truth — never close an RCA without them.

7. **Synthesize.** Report: **root cause entity · affected jobs · temporal progression · fault class · severity ·
   recommended actions**. Worked walkthrough: [references/worked-example.md](references/worked-example.md). Query
   templates: [references/investigation-queries.md](references/investigation-queries.md).

### Rules

1. **Rank by `influencer_score`, not `anomaly_score`, for "which entity?"** — bucket score is aggregate; influencer
   score attributes cause.
2. **Multi-job entities are prime suspects; single-job entities are usually victims.**
3. **Earliest anomaly timestamp wins** — reconstruct chronology from record timestamps across jobs.
4. **`multi_bucket_impact ≥ 3` = sustained behavioral shift**, weight higher than transient spikes.
5. **Use low score thresholds (25 or lower) for influencer/record queries** — high thresholds miss correlated entities.
6. **Never close an RCA without source evidence** from the datafeed index.

From elastic/agent-skills, Apache-2.0.

More from elastic/agent-skills