observability-onboarding

Files (5)

• SKILL.md
• dotnet.md · references/
• java.md · references/
• php.md · references/
• python.md · references/

SKILL.md (first lines)

# Observability Onboarding

Instrument an application with the Elastic Distribution of OpenTelemetry (EDOT) and prove the telemetry arrived. The
instrumentation change is only half the job: an application can be configured perfectly and still emit nothing, so this
skill ends by querying Elasticsearch for the service's data rather than declaring success from a config diff.

The scope is application instrumentation with the EDOT SDKs. Deploying an EDOT Collector, collecting logs from files or
infrastructure, and onboarding data through Elastic Agent, Fleet, or integration packages are separate ingest paths that
this skill does not cover.

Once telemetry is flowing, use the **observability-sre-triage** skill to assess service health,
**observability-k8s-investigation** for Kubernetes-layer failures, **observability-service-reliability** to define SLOs
and alerts on the new signals, and **observability-llm-obs** for GenAI and agentic workloads.

<!-- begin-partial: preamble -->

## Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the
[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
directly.

<!-- end-partial: preamble -->

### Analysis without cluster access

The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the
evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason
from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific
query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is
useful without a cluster; one that only asks for setup is not.

Only the verification step in this skill talks to Elasticsearch; it runs through `POST /_query`. Everything else edits
application code and configuration in the user's workspace.

## Jobs to be done

- Decide whether a service needs fresh instrumentation or a migration off the classic Elastic APM agent
- Apply the EDOT configuration for the service's language and runtime
- Point telemetry at the correct OTLP destination and reject APM Server URLs
- Verify that traces, metrics, and logs land in Elasticsearch under the expected `service.name`
- Diagnose an instrumented service that is reporting nothing, or reporting from the wrong agent

## Output discipline

Applies to every response produced under this skill.

- **Make the edit.** When the language, the current agent, and the destination are known, apply the change. Do not
  narrate a plan and ask for approval on each file; ask only when a decision genuinely cannot be inferred.
- **Do not dump documentation.** Link the relevant page and state the two or three rules that apply to this service.
  Pasting a whole setup guide into the reply is a defect.
- **Verify before reporting success.** "Configured correctly" is not the finding; "telemetry is arriving" is. If
  verification returns no rows, say the service is not reporting and give the most likely cause.
- **Do not speculate past the evidence.** If the verification query is empty, that is one fact with several possible
  causes — name the likeliest and how to distinguish it, rather than asserting one.
- **End on the result.** No trailing offers such as "want me to set up dashboards next?". Follow-up work belongs in a
  recommendations list, phrased as a recommendation.

## Process: route the request

1. **Identify the language and runtime.** Read the project's dependency manifest — `pom.xml` or `build.gradle`,
   `requirements.txt` or `pyproject.toml`, `*.csproj`, `composer.json`. The decision: which language reference to open.
   Also note how the process starts (a Dockerfile `ENTRYPOINT`, a Kubernetes pod spec, a systemd unit), because that is
   where the agent attaches and where the environment variables must be set.

   | Runtime | Reference                                    | How EDOT attaches                                          |
   | ------- | -------------------------------------------- | ---------------------------------------------------------- |
   | Java    | [references/java.md](references/java.md)     | `-javaagent:` flag or `JAVA_TOOL_OPTIONS`                  |
   | Python  | [references/python.md](references/python.md) | `opentelemetry-instrument` entrypoint wrapper              |
   | .NET    | [references/dotnet.md](references/dotnet.md) | `builder.AddElasticOpenTelemetry()` in startup             |
   | PHP     | [references/php.md](references/php.md)       | native extension via OS package, then full process restart |

   For a language not listed, follow the same shape — the required configuration and the following verification step are
   language-independent — and use the upstream EDOT SDK documentation for the attach mechanism.

2. **Detect a classic Elastic APM agent.** The decision: instrument path or migrate path. Each language reference lists
   the markers; the common ones are `ELASTIC_APM_*` environment variables and a language-specific Elastic APM package.
   If any marker is present, take the migrate path — do not layer EDOT on top.

3. **Resolve the telemetry destination.** The decision: which URL goes in `OTEL_EXPORTER_OTLP_ENDPOINT`. Two valid
   answers, and one common wrong one:
   - The **Elastic managed OTLP endpoint** for the deployment or Serverless project. Correct default.
   - An **EDOT Collector** the user already runs, when telemetry must be enriched, sampled, or fanned out before it
     reaches Elastic.
   - **Never an APM Server URL.** If the candidate contains `apm-server`, ends in port 8200, or has an
     `/intake/v2/events` path, it is the classic ingest endpoint and EDOT cannot use it. On the migrate path this is the
     single most common mistake — the old `ELASTIC_APM_SERVER_URL` value must not be carried over.

4. **Apply the shared configuration below, then the language reference.** The shared rules are the same for every
   runtime; the reference covers only what is language-specific.

5. **Verify.** Do not report success from the configuration alone — run the verification process.

## Required configuration

Identical across every EDOT SDK. Set exactly three environment variables:

| Variable                      | Value                                                                                                                 |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| `OTEL_SERVICE_NAME`           | The service's name. This becomes `service.name` in Elasticsearch and is how every other Observability skill finds it. |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | The managed OTLP endpoint or EDOT Collector URL resolved in step 3.                                                   |
| `OTEL_EXPORTER_OTLP_HEADERS`  | The credential, in `Authorization=ApiKey <key>` or `Authorization=Bearer <token>` form.                               |

Two rules that hold for every language:

- **Do not set `OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, or `OTEL_LOGS_EXPORTER`.** The defaults are already
  correct. Setting them is the usual reason a service reports traces but no metrics or logs.
- **Never run a classic Elastic APM agent and EDOT together** on the same process. They double-instrument and produce
  inconsistent traces. Removal and replacement belong in the same change.

## Process: verify telemetry arrives

From elastic/agent-skills, Apache-2.0.

More from elastic/agent-skills