otel-profiles

Files (7)

• SKILL.md
• evals.json · evals/
• backends.md · references/
• kubernetes.md · references/
• nested-containers.md · references/
• receiver.md · references/
• troubleshooting.md · references/

SKILL.md (first lines)

# OpenTelemetry Profiles

The main OpenTelemetry producer is the [eBPF profiler](https://github.com/open-telemetry/opentelemetry-ebpf-profiler), shipped as the `profiling` receiver in the [`otelcol-ebpf-profiler`](https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-ebpf-profiler) Collector distribution. It profiles every process on a Linux host with no application changes.

Maturity: the signal entered [public Alpha](https://opentelemetry.io/blog/2026/profiles-alpha/) in March 2026, the OTLP profiles proto (`v1development`) is Development, the `profiling` receiver is Development, and the Collector gate `service.profilesSupport` is alpha. Expect breaking changes; pin the profiler, Collector, and backend versions together.

Collector-side component facts (OTLP receiver/exporter profile support, the fixed `/v1development/profiles` HTTP path, `k8s_attributes`, `memory_limiter`, `filter` `profile_conditions`) live in the `otel-collector` skill's component pages; OTTL `profile`/`profilesample` contexts live in `otel-ottl`.

`otelcol-ebpf-profiler` `0.163.2` bundles only: receivers `profiling`, `nop`; processors `memory_limiter`, `k8s_attributes`, `resource_detection`, `resource`, `transform` (plus `batch`, which does not support profiles); exporters `otlp_grpc`, `otlp_http`, `debug`, `file`, `nop`; extensions `health_check`, `pprof`, `opamp`, `offcpu`. For anything else (`filter`, the `basicauth` extension, vendor exporters), relay to a regular Collector or build a distribution (`otel-collector-builder` skill).

## Version matrix (Verified 2026-10-25)

| Artifact | Version | What it means |
|---|---|---|
| `otel/opentelemetry-collector-ebpf-profiler` | `0.163.0`, `0.163.1`, `0.163.2` (newest release at verification, 2026-10-21) | Pins ebpf-profiler `v0.0.202640`: `pid_namespace_translation_mode` ([#1801](https://github.com/open-telemetry/opentelemetry-ebpf-profiler/pull/1801)); bundles the `offcpu` extension. Components unchanged from `0.163.0`. |
| Same image, `0.162.0` | last release before the change | Pins ebpf-profiler `v0.0.202636`: boolean `pid_namespace_translation`; does not bundle `offcpu`. |

Evidence: [Docker Hub tags](https://hub.docker.com/r/otel/opentelemetry-collector-ebpf-profiler/tags), the [`v0.163.0`](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.0/distributions/otelcol-ebpf-profiler/manifest.yaml), [`v0.163.1`](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.1/distributions/otelcol-ebpf-profiler/manifest.yaml), and [`v0.163.2`](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.2/distributions/otelcol-ebpf-profiler/manifest.yaml) distribution manifests; the [`v0.163.2` changelog](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.2/CHANGELOG.md).

Read [references/receiver.md](references/receiver.md) when setting keys beyond `samples_per_second`, enabling off-CPU or OBI correlation, or pinning a release not in this table (keys follow the pinned ebpf-profiler, not the Collector version).

## Workflow

1. **Pick the image and pin.** Use the latest release unless a needed key or component is missing from it (see the matrix). Alternative: an OCB build (`otel-collector-builder` skill) based on the [distribution manifest](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/main/distributions/otelcol-ebpf-profiler/manifest.yaml).
2. **Pick the environment.**
   - Linux host or plain Docker: [Run on a host](#run-on-a-host) below.
   - Kubernetes: [references/kubernetes.md](references/kubernetes.md) (DaemonSet, RBAC, Kubernetes metadata, memory sizing).
   - Kubernetes nodes that are themselves containers (kind, k3d, Docker-in-Docker, minikube with the docker driver): also [references/nested-containers.md](references/nested-containers.md).
3. **Wire the pipeline.** `profiles` pipeline: `profiling` receiver, `memory_limiter` first, enrichment processors, then `otlp_grpc` or `otlp_http`. Start the Collector with `--feature-gates=+service.profilesSupport`. Run `validate --config=<file> --feature-gates=+service.profilesSupport` with the target image; it catches wrong keys for the pinned version.
4. **Export and verify.** Add `debug` (`verbosity: basic`) first: each `reporter_interval` it logs `"resource profiles": N, "sample records": M`; no lines or zero counts mean the receiver is not producing (see troubleshooting). `verbosity: detailed` shows resource attributes. Then add the backend: [references/backends.md](references/backends.md) for endpoints, auth, and per-backend label behavior.
5. **Troubleshoot** by exact error string: [references/troubleshooting.md](references/troubleshooting.md).

## Run on a host

The profiler needs root or `CAP_SYS_ADMIN`/`CAP_BPF`/`CAP_PERFMON`, the host PID namespace, and a mounted tracefs. A container gets a fresh `/sys` without tracefs even when privileged, so bind-mount it:

```bash
docker run -d --name otel-ebpf-profiler --privileged --pid=host \
  -v /sys/kernel/tracing:/sys/kernel/tracing \
  -v "$PWD/config.yaml:/etc/otelcol/config.yaml:ro" \
  otel/opentelemetry-collector-ebpf-profiler:0.163.2 \
  --config=/etc/otelcol/config.yaml --feature-gates=+service.profilesSupport
```

```yaml
receivers:
  profiling:
    samples_per_second: 20      # default
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 800
    spike_limit_mib: 200
exporters:
  debug:
    verbosity: basic
  otlp_grpc:
    endpoint: profiles-backend:4317   # backend-specific, see references/backends.md
    tls:
      insecure: true
service:
  pipelines:
    profiles:
      receivers: [profiling]
      processors: [memory_limiter]
      exporters: [otlp_grpc, debug]
```

The image is built `FROM scratch`: no shell, so debug from logs, not `exec`. Expected startup lines: `eBPF tracer loaded`, `Attached sched monitor`, `Everything is ready`.

## Known limitations

- The receiver profiles every process; it has no process, container, or namespace selection, and upstream declined to add one ([#358](https://github.com/open-telemetry/opentelemetry-ebpf-profiler/issues/358)). To keep a subset, filter on enrichment attributes with `filter` `profile_conditions`; see "Keep only some workloads" in [references/kubernetes.md](references/kubernetes.md). Sampling cost on the node stays the same; only exported volume drops.
- Linux `amd64`/`arm64`, kernel 5.10+ (checked at config validation; `no_kernel_version_check` bypasses it). Backends may lag the proto; check which OTLP profiles version they accept.
- Trace-to-profile correlation is partial: OTLP profiles can link samples to `trace_id`/`span_id`, but the profiler fills them only from a context producer (OBI via `obi_process_ctx`; the [thread-context OTEP](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.62.0/oteps/profiles/4947-thread-ctx.md) is accepted, but the receiver's `thread_context` support in `v0.0.202640` is a stub). Go pprof labels are read as custom labels.

From ollygarden/opentelemetry-agent-skills, Apache-2.0.

More from ollygarden/opentelemetry-agent-skills