production-observability

Files (1)

• SKILL.md

SKILL.md (first lines)

# production-observability

The bar: an on-call engineer who has never seen this code can tell that it is
failing, find the one request, follow it across every service and queue it
touched, and see why, from telemetry alone. The telemetry must not leak data,
cost more than it earns, or take the app down.

Each item below has an id. Use the same ids as the rules your code review
checks, so the standard you build to is the standard each change is reviewed
against.

## The standard

| Id                    | Done when                                                                                                                                                                                          | With autotel                                                                                                                             |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `safe-telemetry`      | Init runs once, before the code it instruments, and cannot crash the app. Service name, version and environment come from config. Telemetry flushes on every exit.                                 | `instrumentation.ts` loaded with `--import`; `init({ service, version, environment })`; `processHandlers: true` or your own `shutdown()` |
| `boundary-signal`     | Every route, consumer, job and CLI command has a span, and every outbound call (HTTP, DB, cache, queue, model API) has a child span.                                                               | Framework adapter or middleware; `autoInstrumentations`; `trace()` on jobs; `traceConsumer` / `traceProducer` from `autotel/messaging`   |
| `context-propagation` | One trace spans every service and queue a request touches. Deferred work stays inside the span that started it.                                                                                    | `http` / `undici` instrumentation for outbound calls; `traceProducer` injects context into messages and `traceConsumer` continues it     |
| `request-context`     | Each request's wide event carries the bounded ids that find it (tenant, user, order or job id, outcome) and which way each decision went (branch, fallback, retry, cache, flag variant, provider). | `getRequestLogger().set({...})`, one wide event per request                                                                              |
| `structured-logs`     | Server code logs through one structured logger, and every line carries `trace_id` and `span_id`.                                                                                                   | `autoInstrumentations: ['pino']` plus `logs: true`, with the logger created after `init()`                                               |
| `error-context`       | Errors keep their cause, mark the span ERROR, and are logged once with ids.                                                                                                                        | `createStructuredError({ message, why, fix, cause })` thrown through `trace()`, or `recordStructuredError(ctx, err)`                     |
| `alertable-signal`    | Every dependency, queue and job emits outcome by bounded reason, a duration, and for queues the lag. Alerts and SLOs read them.                                                                    | `Metric#trackOutcome`, `createHistogram`, `createObservableGauge`, `autotel/slo` (skill `design-alertable-metrics`)                      |
| `feature-signal`      | A user-facing change emits the event or metric that shows whether it worked.                                                                                                                       | `track()` or `Event` (skill `autotel-events`)                                                                                            |
| `no-sensitive-data`   | No secrets, tokens, cookies, whole bodies or headers, or personal data beyond agreed ids reach telemetry.                                                                                          | `attributeRedactor: 'default'` set explicitly; fields chosen, never spread                                                               |
| `bounded-attributes`  | Span names and metric labels are bounded: route templates, enum reasons. Ids live on spans and logs only.                                                                                          | `spanNameNormalizer: 'rest-api'`; labels from closed sets                                                                                |
| `silent-drop`         | Skipped, filtered, retried-out and fallen-back work is counted with a reason.                                                                                                                      | A counter or a `log.warn` with the reason, inside the span                                                                               |
| `telemetry-config`    | Sampling keeps every error and slow request; health checks and probes are filtered out; production runs at the agreed log level.                                                                   | `sampling: 'production'` set explicitly (unset keeps everything); `spanFilter` for probes (skill `tune-sampling`)                        |
| `telemetry-contract`  | Names that dashboards and alerts read are treated as a contract and covered by a test.                                                                                                             | `createTraceCollector()` from `autotel/testing`, or `autotel-vitest`                                                                     |

Ownership of the dashboards and alerts (skill `autotel-grafana`) and audit
trails for money and auth paths (skill `build-audit-trails`) build on this. They
are not part of the minimum.

## Existing codebase

Work in this order. Each step is mergeable on its own, and the earlier ones
make the later ones visible.

1. **Measure.** Run `npx autotel-cli map --json --no-write` (Node 22 or later)
   and keep the score and the dark routes for the report. On an unsupported
   framework, list entry points by hand.
2. **Startup and shutdown (`safe-telemetry`).** Set up the instrumentation file
   (see [New project](#new-project)) and run the app with it. Then check the
   points below before moving on:
   - With no endpoint configured, autotel exports nothing.
   - In production, set `OTEL_EXPORTER_OTLP_ENDPOINT` to the base URL. autotel
     appends `/v1/traces`, `/v1/metrics` and `/v1/logs`, so a URL that already
     ends in `/v1/traces` breaks metrics.
   - `init()` throws on an unknown `sampling` preset or a bad
     `attributeRedactor`, so validate config before deploying.
   - `beforeExit` does not fire on `process.exit()`. Code that exits itself must
     `await shutdown()` first.
3. **Entry points and outbound calls (`boundary-signal`, `context-propagation`).**
   - Add the framework adapter (skill `autotel-frameworks`).
   - Put the libraries the service calls in `autoInstrumentations`. It is an
     allowlist and is off by default in Node, so list `http` for outbound
     calls.
   - Wrap jobs, crons, consumers and CLI commands in `trace()` or
     `traceConsumer`.
   - Re-run `map` until no route is dark.
4. **Request context and logs (`request-context`, `structured-logs`).**
   - In each handler, call `getRequestLogger().set()` with the ids that find
     the request.
   - Route the existing logger through instrumentation so every line carries
     the trace.
   - Replace `console.*` in server code.
   - Skill `autotel-request-logging` covers this step.
5. **Errors (`error-context`).**
   - Replace bare `throw new Error('failed')` and empty catches with structured
     errors that keep `cause`.
   - At the HTTP boundary, return `parseError(err)`, which strips `internal`.
   - Skill `autotel-structured-errors` covers this step.
6. **Data safety and cost (`no-sensitive-data`, `bounded-attributes`, `telemetry-config`).**
   - Set `attributeRedactor` explicitly. It is automatic only when the
     resolved environment is `production`, and the environment defaults to
     `NODE_ENV`.
   - Set `spanNameNormalizer` and filter health probes with `spanFilter`.
   - Set `sampling: 'production'` explicitly. It keeps every error and slow
     call and 10% of the rest. With no `sampling` at all, nothing is dropped,
     which is right in development and expensive at scale. With your own
     `spanProcessors`, wrap the exporting one in `TailSamplingSpanProcessor`
     (`init()` warns when none is). Skill `tune-sampling` covers what it does
     and doesn't evaluate.
7. **Signals an alert can use (`alertable-signal`, `silent-drop`).** For each
   dependency, queue and job:
   - count outcomes by a bounded reason
   - record a duration
   - for queues, observe the lag

   Skill `design-alertable-metrics` turns these into alerts, and
   [`incident-readiness.md`](../find-observability-gaps/references/incident-readiness.md)
   lists what to emit per area.

8. **Prove it (`telemetry-contract`).**
   - Add a test per critical path that asserts the span, its key attributes
     and its error status (see [Proving it](#proving-it)).
   - Run the app against `autotel-devtools` and follow one real request end to
     end.
9. **Lock it in.** Set up [enforcement](#enforcement), so the next PR cannot
   undo the work.

Leave a product decision to the team rather than guessing: which ids count as
personal data, SLO targets, and which vendor to use. Name each one in the
report.

## New project

Start with the instrumentation file and the shutdown path, before the first
route, so every later file is written against them.

```typescript
// src/instrumentation.ts — loaded with: node --import ./dist/instrumentation.js dist/server.js
import 'autotel/register'; // ESM loader hook; must come first
import { init } from 'autotel';

init({
  service: process.env.OTEL_SERVICE_NAME ?? 'orders-api',
  // version: SERVICE_VERSION, DD_VERSION or APP_VERSION, else the deployed
  // commit (GIT_SHA, GITHUB_SHA, ...), else package.json

From jagreehal/autotel, Apache-2.0.

More from jagreehal/autotel