elasticsearch-anomaly-detection

Files (2)

• SKILL.md
• anomaly-detection-reference.md · references/

SKILL.md (first lines)

# Elasticsearch Anomaly Detection

Create, open, and start ML anomaly detection jobs on time-series data. Choose the right count-family detector direction,
configure bucket span and time field, wire the datafeed to the correct index, and confirm running state from stats — not
from assumptions.

<!-- begin-partial: preamble -->

## Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the
[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
directly.

<!-- end-partial: preamble -->

> **Prerequisite:** ML anomaly detection requires a Platinum-equivalent license on self-managed clusters. Serverless
> projects include ML. The caller needs `manage_ml` to create and manage jobs.
>
> **Related skill:** For interpreting anomaly scores, influencers, and model behavior after a job is running, use
> `elasticsearch-anomaly-detection-explainer` — not this skill.

## Process

1. **Discover the target index and time field.** List candidate indices with `GET /_cat/indices` (pass a pattern when
   the user names one). Fetch field types for the chosen index with `GET /{index}/_mapping`. The decision: confirm the
   index exists, identify the time field (often `@timestamp`), and verify document volume is sufficient for baseline
   learning. Never guess index or field names — they vary across deployments.

2. **Choose detector function and direction.** Match the user's intent to a count-family detector in
   `analysis_config.detectors`:
   - **Spike, surge, unusual increase in event volume** → `high_count` (or `count`, which flags both directions but is
     acceptable when the user cares about spikes). Do **not** use `low_count` — it will miss spikes.
   - **Drop, outage, absence of events, traffic stops** → `low_count`. Do **not** use `high_count` — it will miss drops
     and silence.
   - **Metric deviation** (CPU, latency, a numeric field) → mean-family functions (`mean`, `high_mean`, `low_mean`) with
     `field_name` set — only when the user asks about a numeric metric, not raw event volume.

   The decision: pick one primary detector whose direction matches the anomaly type. For volume spike/drop questions on
   document counts, stay in the count family — mean detectors are unsuited to "how many events" questions.

3. **Set immutable job shape before creation.** These fields cannot change after `PUT /_ml/anomaly_detectors/{job_id}`:
   - `analysis_config.bucket_span` — use the interval the user specifies (e.g. `15m` for 15-minute buckets). Match the
     granularity of anomalies they care about; too short is noisy, too long is slow to detect.
   - `data_description.time_field` — the time field from the mapping (commonly `@timestamp`).
   - `analysis_config.detectors` — the function and direction from step 2.

   Example job body for a volume-spike detector:

   ```json
   {
     "analysis_config": {
       "bucket_span": "15m",
       "detectors": [{ "function": "high_count" }]
     },
     "data_description": { "time_field": "@timestamp" }
   }
   ```

   Example for an outage / drop detector:

   ```json
   {
     "analysis_config": {
       "bucket_span": "15m",
       "detectors": [{ "function": "low_count" }]
     },
     "data_description": { "time_field": "@timestamp" }
   }
   ```

4. **Create the job.** Call `PUT /_ml/anomaly_detectors/{job_id}` with the job id the user requested (or a descriptive
   id you propose). The job starts in `closed` state — creating it does not start analysis.

5. **Create the datafeed.** Call `PUT /_ml/datafeeds/datafeed-{job_id}` immediately after job creation. Set `job_id` to
   the same id, `indices` to the target index (exact name or pattern from step 1), and a query that selects the relevant
   documents (typically `match_all`). The datafeed id convention is `datafeed-{job_id}`.

   ```json
   {
     "job_id": "{job_id}",
     "indices": ["{index}"],
     "query": { "match_all": {} }
   }
   ```

6. **Open the job, then start the datafeed — in that order.** This sequence is mandatory; do not skip or reorder:
   1. `POST /_ml/anomaly_detectors/{job_id}/_open` — transitions the job to `opened`.
   2. `POST /_ml/datafeeds/datafeed-{job_id}/_start` — transitions the datafeed to `started`.

   Opening before the datafeed exists fails. Starting the datafeed before opening the job fails. Do not report success
   after only creating resources — the job is not running until both are active.

7. **Confirm running state from stats.** Verify the outcome with:
   - `GET /_ml/anomaly_detectors/{job_id}/_stats` — expect `state: "opened"`.
   - `GET /_ml/datafeeds/datafeed-{job_id}/_stats` — expect `state: "started"`.

   Optionally call `GET /_ml/anomaly_detectors/{job_id}` to confirm configuration (detectors, `bucket_span`,
   `time_field`, datafeed indices). Report both stats states explicitly — "created" is not the same as "opened" and
   "started".

## Teardown

When stopping or deleting a job, reverse the startup order:

1. `POST /_ml/datafeeds/datafeed-{job_id}/_stop` — stop the datafeed first.
2. `POST /_ml/anomaly_detectors/{job_id}/_close` — then close the job.

Stop the datafeed before closing the job. Close the job before resetting or deleting it.

## Guidelines

- **Required lifecycle order (create):** job → datafeed → open job → start datafeed. Every new job follows this
  sequence.

From elastic/agent-skills, Apache-2.0.

More from elastic/agent-skills