go-observability

Files (5)

• SKILL.md
• alerting.md · references/
• metrics.md · references/
• slog.md · references/
• tracing.md · references/

SKILL.md (first lines)

# Go Observability

Observability is not optional for production services. Every service must produce
structured logs, expose metrics, and propagate trace context. Use the stdlib
`log/slog` for logging and OpenTelemetry for tracing and metrics.

> Community default. A project skill or agent-config rule that explicitly
> supersedes `go-observability` takes precedence over this one.

Detailed reference material, loaded on demand:

- `references/slog.md` — handler setup, logger injection, child loggers,
  `LogValuer` redaction, trace-ID handler, hot paths, runtime levels.
- `references/tracing.md` — tracer provider, spans, propagation, sampling,
  profile labels, shutdown.
- `references/metrics.md` — metric definitions, HTTP instrumentation.
- `references/alerting.md` — RED/USE, histograms vs summaries, burn-rate
  alerts, alert hygiene, exemplars.

Read a reference file only when the section below is not enough.

## 1. Structured Logging with slog

Use `log/slog` (Go 1.21+) with a JSON handler in production. Every line is
key-value pairs, never a formatted sentence:

```go
// ✅ Good — structured, leveled
logger.Info("user created", slog.String("user_id", user.ID), slog.Duration("latency", elapsed))

// ❌ Bad — unparseable in production
log.Printf("user %s created in %v", user.ID, elapsed)
```

Inject the logger as a dependency; never reach for a package-level global.
Derive child loggers with `logger.With(...)` so component, method and request
ID are attached once instead of at every call site. See `references/slog.md`.

### Log levels — use them consistently

| Level | Use for |
|---|---|
| `Debug` | Verbose diagnostic info, disabled in production |
| `Info` | Normal operations: request received, job completed |
| `Warn` | Recoverable issues: retry succeeded, deprecated usage |
| `Error` | Failures requiring attention: DB down, external call failed |

NEVER log at Error level for expected conditions (user not found → Info or Warn).

### Sensitive data — NEVER log

- Passwords, tokens, API keys
- Full credit card numbers, SSNs
- Personal identifiers: email addresses, phone numbers, IP addresses
  where they identify a person
- Raw request bodies containing PII

```go
// ✅ Good — an opaque internal ID, no personal data
logger.Info("auth attempt", slog.String("user_id", userID), slog.Bool("success", ok))

// ❌ Bad — leaks a credential and a personal identifier
logger.Info("auth attempt", slog.String("password", password))
logger.Info("auth attempt", slog.String("user", email))
```

Log the stable internal ID and join to personal data in the system that is
allowed to hold it. Log retention, search access and third-party log sinks
rarely meet the bar that applies to the user table.

Make redaction a property of the type: implement `slog.LogValuer` on
secrets and on structs that hold personal data, so `slog.Any(user)` logs
only the safe fields everywhere.

### Cost and control

- Hot paths use `logger.LogAttrs` and guard expensive attributes with
  `logger.Enabled`.
- A `slog.LevelVar` in `HandlerOptions` lets the level change at run time
  without a deploy.
- `slog.NewMultiHandler` (Go 1.26+) fans out to several handlers.

## 2. Distributed Tracing with OpenTelemetry

Start a span for every operation worth seeing on a waterfall — inbound
requests, DB calls, outbound HTTP, meaningful business steps — and always
`defer span.End()`.

Rules the reference examples follow:

- Name the span after the operation (`GetUser`, `db.query`), never after the
  fully-qualified function, and never `doStuff`.
- On failure call `span.RecordError(err)` and `span.SetStatus(codes.Error, ...)`,
  or the trace shows a green span for a failed request.
- Pass `ctx` down the whole chain. A `context.Background()` mid-chain silently
  starts a new trace and breaks the parent link.
- Sample with `ParentBased(TraceIDRatioBased(r))` so services agree on a
  trace; keep errors and slow requests with tail sampling in the collector.
- Profiles are the fourth signal: continuous profiling in production, with
  bounded `pprof.Labels` (route, tenant tier) to split a hot function.

Setup and examples in `references/tracing.md`.

## 3. Metrics with OpenTelemetry / Prometheus

| Type | Use for | Example |
|---|---|---|
| Counter | Monotonically increasing values | Requests total, errors total |
| Gauge | Values that go up and down | Active connections, queue depth |
| Histogram | Distribution of values | Request latency, response size |

Naming is `<namespace>_<subsystem>_<name>_<unit>`:

```text
http_request_duration_seconds     ✅ (unit in name)
http_requests_total               ✅ (counter with _total suffix)
db_connections_active             ✅ (gauge, no suffix needed)
user_signups                      ❌ (missing _total for counter)
requestLatency                    ❌ (camelCase, no unit)

From eduardo-sl/go-agent-skills, MIT.

More from eduardo-sl/go-agent-skills