debug-missing-spans

Files (1)

• SKILL.md

SKILL.md (first lines)

# Debug missing spans

When a span you expect isn't in the backend, the cause is somewhere in this chain:

```
code → SDK init → sampler → processor → exporter → network → backend ingest → backend index
```

This skill walks each link in order with a quick check you can run. Don't skip steps. The cause is rarely where you'd guess.

## Check these three first

1. **No endpoint, no export.** Node `init()` has no default endpoint and no
   localhost fallback. Without `endpoint`, `destinations`, `spanExporters`,
   `spanProcessors` or `OTEL_EXPORTER_OTLP_ENDPOINT`, no exporter is built and
   every span is silently discarded. Set one:

   ```typescript
   init({ service: 'my-app', endpoint: 'http://localhost:4318' });
   ```

   `endpoint` is the base URL; autotel appends `/v1/traces`, `/v1/metrics`
   and `/v1/logs`.

2. **Sampling.** With `sampling: 'production'` (or
   `samplingPresets.production()`), about 90 % of fast, successful spans from
   `trace()` / `withTracing()` / `instrument()` are dropped on export, and each
   wrapped span decides on its own. Reproduce with `sampling: 'development'`.
   See [`tune-sampling`](../tune-sampling/SKILL.md).

3. **Two copies of `@opentelemetry/api` (Next.js, bundlers).** An
   instrumentation on a different API copy than autotel sends its spans to a
   provider nobody exports from. `init()` warns and names it. Under Next.js,
   use `withAutotelConfig` from `autotel-adapters/next-config` and add
   `@opentelemetry/api` to the app's dependencies.

## Step 0: Reproduce locally with the pretty exporter

Before chasing remote backends, confirm the span exists at all:

```typescript
init({
  service: 'my-app',
  debug: 'pretty', // hierarchical colourised output to stdout
  sampling: 'development', // keep everything while reproducing
});
```

The pretty exporter prints one tree per trace when the trace's root span ends; a trace whose root never exports prints after two idle seconds, and `flush()` prints anything still waiting. If you see the span in stdout, the code and sampler are fine. Skip to "exporter / network". If you don't, keep reading.

## Step 1: Is the SDK actually initialised?

Common failure: `init()` runs after the code you expect to trace, or after the
library you expect auto-instrumented was imported.

```typescript
import { isInitialized } from 'autotel';

console.log('[autotel-debug] initialised:', isInitialized());
```

If `false`, `init()` has not run. Move it to the very top of the entry file (or to `instrumentation.ts` for Next.js). Wrapping a function with `trace()` at module load is fine; calling it before `init()` is not.

Auto-instrumented spans (HTTP, pg, pino, …) need three things: the library
listed in `autoInstrumentations` (it is off by default), the
`@opentelemetry/auto-instrumentations-node` package installed, and `init()`
running before the library is imported. Under ESM, start node with
`--import autotel/register`. `init()` prints a console warning for a missing
loader hook or a logger loaded too early.

## Step 2: Sampler

- `sampling: 'production'` / `'errors-only'` drop fast, successful wrapped
  spans; `'off'` drops all of them.
- A `sampler` passed to a single `trace()` / `withTracing()` / `instrument()`
  overrides the one from `init()`.
- With neither `sampling` nor `sampler`, `OTEL_TRACES_SAMPLER` /
  `OTEL_TRACES_SAMPLER_ARG` from the environment set the SDK head sampler.

To keep one request whatever the sampler says, call `forceKeep()` in its span,
or send `baggage: autotel.debug=1` (read where incoming context is extracted).

Edge (`autotel-edge` / `autotel-cloudflare`): check `sampling.headSampler`
(a `{ ratio }` stops spans being recorded) and `sampling.tailSampler`.
`getActiveConfig()` from `autotel-edge` returns the resolved edge config.

## Step 3: Cloudflare Workers: `ctx.waitUntil`

The single biggest cause of missing spans on the edge: **the response returned before the exporter flushed**.

If you're using `addEventListener('fetch', …)` or a hand-rolled `fetch` in a module worker without wiring `ctx.waitUntil(…)` to the export call, async drains drop silently. `autotel-edge` also warns once when the config has no `exporter` and no `spanProcessors`.

Fix. Switch to `defineWorkerFetch` or `wrapModule`, both of which wire `waitUntil` automatically:

```typescript
import { defineWorkerFetch } from 'autotel-cloudflare';

export default defineWorkerFetch(
  {
    service: { name: 'edge' },
    exporter: { url: 'https://otlp.example.com/v1/traces' },
  },
  async (request, env, ctx, log) => {
    // log.set / spans here all flush via ctx.waitUntil before response returns
    return new Response('ok');
  },
);
```

## Step 4: Processor pipeline

Read the `init()` call, not the provider internals. Common issues:

- **`spanFilter` returns `false` for your span.** It wraps every processor.
- **A `FilteringSpanProcessor` you added drops it.** Its `filter` must return `true` to keep.
- **`TailSamplingSpanProcessor` dropped it** (fast, successful, not force-kept).
- **`spanProcessors: []`** is an explicit off switch: nothing is exported.
- **Edge only: a `postProcessor` / `composePostProcessors` step returns `[]` for your span.**

From jagreehal/autotel, Apache-2.0.

More from jagreehal/autotel