elasticsearch-reindex

Files (5)

• SKILL.md
• api-reference.md · references/
• patterns.md · references/
• troubleshooting.md · references/
• tuning.md · references/

SKILL.md (first lines)

# Elasticsearch Reindex

Copy documents from source indices or data streams to a destination using `POST /_reindex`. An expert reindex workflow
prepares the destination explicitly, chooses local versus remote execution, filters at the source when only a subset is
needed, runs long copies asynchronously, tracks the task to completion, and verifies the destination document count
before reporting results.

<!-- begin-partial: preamble -->

## Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the
[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
directly.

<!-- end-partial: preamble -->

## Process

1. **Confirm connectivity and deployment type.** Call `GET /`. Read `build_flavor` and `version.number` to know whether
   shard, replica, and cluster-settings APIs are available (Serverless manages shards/replicas internally and blocks
   most `_cluster/*` APIs). The decision: continue only when the cluster is reachable. If the call fails, stop — do not
   guess endpoints or credentials.

2. **Decide local versus remote reindex.** Compare where the source and destination live.
   - **Same cluster** — use local reindex: `source.index` and `dest.index` only. Do **not** add `source.remote` when
     both indices are on the cluster you are connected to.
   - **Different cluster** — use reindex from remote: add `source.remote` with the remote cluster URL and credentials.
     Remote reindex does not support slicing; compensate with query-based partitioning (date ranges, term filters)
     across parallel requests. Confirm the remote host is allowlisted on Self-Managed / ECH (`reindex.remote.whitelist`
     in cluster config); Serverless manages allowlisting internally (ECH remotes only, Tech Preview).

   Data needed: source index name(s), destination index name, and whether they share a cluster.

3. **Inspect the source — never guess field names or counts.** Call `GET /{source}/_mapping` to ground field names and
   types. Call `GET /{source}/_count` (or `GET /_cat/count/{source}?h=count` on Self-Managed / ECH) to learn how many
   documents exist.

   The decision: **full copy** versus **filtered subset**.
   - Full copy — omit `source.query` (match-all behavior).
   - Filtered subset — add `source.query` with Query DSL. For time ranges, use a `range` filter on the timestamp field
     (commonly `@timestamp`), e.g. `"gte": "2025-01-01", "lt": "2025-02-01"` for January 2025. Do **not** run a
     full-index copy when the user asked for a date range or other filter.

   Data needed: the user's filter criteria and the mapping-confirmed field names.

4. **Prepare the destination index before copying.** `_reindex` does **not** copy mappings, shard counts, or analyzers.
   Create the destination with explicit settings and mappings derived from the source mapping via `PUT /{dest}`.
   - On Self-Managed / ECH: set `number_of_replicas: 0` and `refresh_interval: "-1"` on the destination during the copy
     for write throughput; restore production values afterward with `PUT /{dest}/_settings`.
   - On Serverless: omit `number_of_shards` and `number_of_replicas` (managed by Elastic); you may set
     `refresh_interval: "-1"` during the copy.
   - For **data stream** destinations: ensure an index template with `data_stream: {}` exists, create the data stream,
     and set `dest.op_type` to `"create"` (append-only).

   The decision: create/prepare the target rather than relying on auto-creation with dynamic mapping. Wrong or missing
   mappings cause partial failures or silent type coercion.

   Data needed: destination name, corrected or compatible mappings, and deployment-specific settings constraints.

5. **Build and submit the reindex request.** Call `POST /_reindex?wait_for_completion=false` for any copy that may take
   more than a few seconds or when the user says the index is large — the response returns a **task id** immediately
   instead of blocking.

   Request body essentials:
   - `source.index` — source index or data stream (correct name, not reversed with `dest.index`).
   - `dest.index` — prepared destination from step 4.
   - `source.query` — include only when step 3 chose a filtered subset.
   - `conflicts: "proceed"` — when retrying a partially complete reindex.
   - Optional tuning: `source.size` (batch size), `requests_per_second` (throttle), `slices=auto` on local reindex only
     (parallelize per primary shard — never for remote), `scroll` (increase keep-alive on slow clusters), `max_docs`
     (test runs), `script` (transform), `dest.pipeline` (ingest enrichment).

   Example filtered subset (January 2025 only):

   ```json
   {
     "source": {
       "index": "eval-reindex-src",
       "query": {
         "range": {
           "@timestamp": { "gte": "2025-01-01", "lt": "2025-02-01" }
         }
       }
     },
     "dest": { "index": "eval-reindex-jan" }
   }
   ```

   Do **not** reach for `_split`, `_shrink`, or snapshot/restore when the task is a filtered subset copy or a straight
   document migration — those APIs solve different problems.

6. **Track the task to completion.** Store the task id from the reindex response. Poll `GET /_tasks/{task_id}` until
   `completed` is `true`. Read `status.total`, `status.created`, and `response.failures`. On Self-Managed / ECH you may
   also list active reindex tasks with `GET /_tasks?actions=*reindex&detailed`; on Serverless, query by task id only
   (list/cancel are not available). Adjust throttling mid-flight with
   `POST /_reindex/{task_id}/_rethrottle?requests_per_second=N` without canceling.

7. **Verify and report the destination count.** Call `GET /{dest}/_count` (works on all deployment types). On
   Self-Managed / ECH you may also use `GET /_cat/count/{dest}?h=count`. Compare source filter expectations to the
   destination count. Report the **exact** count from the destination — do not estimate or guess.

   After a successful full copy, restore production settings on the destination with `PUT /{dest}/_settings` (replicas
   and refresh interval on Self-Managed / ECH; refresh interval only on Serverless).

## Deployment constraints

| Capability                  | Self-Managed / ECH | Serverless                                |
| --------------------------- | ------------------ | ----------------------------------------- |
| Local reindex               | Full support       | Full support                              |
| Reindex from remote         | Full support       | Tech Preview — ECH remotes only           |
| `number_of_shards/replicas` | User-configurable  | Managed — omit on index creation          |
| `slices=auto` (local only)  | Supported          | Supported for local reindex               |
| `GET /_cat/count/{index}`   | Supported          | Not available — use `GET /{index}/_count` |

From elastic/agent-skills, Apache-2.0.

More from elastic/agent-skills