analyze-traces

Files (1)

• SKILL.md

SKILL.md (first lines)

# Analyze traces

This skill teaches an AI assistant how to read and reason about OpenTelemetry traces produced by autotel. Whether they're sitting in a backend, exported to a local JSON dump, or captured in a test.

## When to use

- Debugging a failing endpoint after deploy
- Investigating latency regressions (p50 / p95 / p99 spike)
- Following a single request across browser → server → queue → worker
- Auditing attribute hygiene (cardinality, PII leak risk, noise)
- Spot-checking that a new instrumentation actually produces the spans you expected

## Input formats

| Source                         | How to access                                                      |
| ------------------------------ | ------------------------------------------------------------------ |
| Local debug dump               | `.autotel/spans/*.ndjson`: one span per line, OTLP JSON shape      |
| `InMemorySpanExporter` (tests) | `exporter.getFinishedSpans()`                                      |
| Backend (interactive)          | Jaeger / Tempo / Honeycomb UI; Datadog Trace Search; etc.          |
| Backend (programmatic)         | Honeycomb Query API, Tempo `/api/search`, Datadog Logs / Trace API |

## The shape of an autotel span

```json
{
  "name": "POST /api/checkout",
  "context": { "traceId": "…", "spanId": "…" },
  "parentSpanId": "…",
  "kind": "SERVER",
  "startTimeUnixNano": "…",
  "endTimeUnixNano": "…",
  "status": { "code": "OK" },
  "attributes": {
    "service.name": "checkout",
    "http.request.method": "POST",
    "url.full": "https://api.example.com/api/checkout",
    "http.response.status_code": 200,
    "user.id": "usr_123",
    "user.plan": "enterprise",
    "cart.items": 3,
    "cart.total": 14999,
    "_correlationId": "01J…"
  },
  "events": [
    {
      "name": "log.emit.manual",
      "attributes": { "level": "info", "stage": "validated" }
    }
  ],
  "links": [],
  "resource": {
    "service.name": "checkout",
    "deploy.id": "v2025.05.04-1"
  }
}
```

Key conventions to recognise:

- `service.name` distinguishes services in a multi-service trace.
- `_correlationId` (autotel-specific) is stable within a logical unit of work even across forked child spans (`_parentCorrelationId` ties them).
- `gen_ai.*` attributes follow OpenTelemetry gen-ai semantic conventions for LLM calls.
- `exception.*` attributes (auto-set by `createStructuredError`) carry `type`, `message`, `stacktrace`.

## Common investigations

### "Why is endpoint X failing?"

1. Find the trace: filter `service.name = "<svc>" AND http.route = "<route>" AND status = error` for the last hour.
2. Open the slowest / latest matching trace.
3. Inspect the root span's `exception.message` and `exception.stacktrace`.
4. Walk down the child spans: the deepest span with `status.code = ERROR` is usually the culprit.
5. If using `createStructuredError`, look for `code`, `why`, `internal.*` attributes. They usually answer the "why" without you reading code.

### "Why is endpoint X slow?"

1. Find a slow trace: `service.name = "<svc>" AND http.route = "<route>" AND duration > p99(duration)`.
2. View the waterfall: pinpoint the longest child span by self-time (not wall time).
3. Common offenders:
   - **Sequential awaits that should be parallel**: sibling spans run end-to-end instead of overlapping.
   - **N+1 queries**: many short same-named spans (`SELECT * FROM …`) under one parent.
   - **Cold starts**: `faas.coldstart=true` in Workers or Lambda.
   - **Tool retries**: gen-ai spans with `gen_ai.response.finish_reasons` containing `error` followed by another call.

### "Which requests are affected?"

A waterfall explains one slow request. It cannot tell you what the slow ones
share. Run the core analysis loop instead: split the population on the symptom,
then score every recorded field by how much the two groups disagree.

```typescript
import { compareCohorts } from 'autotel/analysis';

const isSlow = (event) => Number(event['checkout.duration_ms']) >= 800;

const ranked = compareCohorts({
  outlier: events.filter(isSlow),
  baseline: events.filter((event) => !isSlow(event)),
});
// [{ field: 'payment.provider', value: 'bank-beta', difference: 0.67, … }]
```

Feed it any array of flat records: `span.attributes` from a collector, wide
events from `getRequestLogger()`, or rows returned by a backend query.

The loop skips fields whose values never repeat. A request id takes a fresh
value on every event, so it would bury the answer under noise. Bucket durations
and payload sizes when you instrument them if you want the loop to rank them.

The top row is a hypothesis. A field can move with the problem without causing
it. You are done when you have named the cohort and confirmed the mechanism in
two traces drawn from it.

### "Follow this user across services"

Use `_correlationId` (or `user.id` if you have it):

```
service.name in (web, api, worker) AND _correlationId = "01J…"

From jagreehal/autotel, Apache-2.0.

More from jagreehal/autotel