observability

Files (1)

• SKILL.md

SKILL.md (first lines)

# observability

If you can't see it, you can't run it: structured logs, RED/USE metrics, traces, and honest health checks.

## Activate when

- Operating, debugging, or shipping any deployed service; adding alerts, dashboards, or probes.

## Do NOT activate for

- One-off local debugging (use a debugger/log); frontend-only analytics (see product/mvp-discovery, startup/launch-ops).

## Inspection

Inventory existing signals: log format/levels, emitted metrics, trace coverage, health endpoints, alert rules, dashboards. Find the gaps against the failure modes below.

## Decision rules

- Logs: structured (JSON) with trace/request IDs; levels meaningful (ERROR = page someone); no PII/secrets; correlation across service hops.
- Metrics: RED (rate/errors/duration) per endpoint, USE (utilization/saturation/errors) per resource; business metrics only where they drive alerts.
- Tracing: OpenTelemetry spans across service/queue boundaries; sample head- or tail-based deliberately.
- Health: liveness (restart me) vs readiness (send traffic) as separate probes hitting real checks; graceful shutdown drains first.
- Alerts: symptom-based, runbook-linked, with halt/escalation criteria — never alert on what nobody will act on.

## Procedure

1. Close the highest-risk gap first (usually: request IDs, error-rate alert, readiness probe).
2. Wire one dashboard per service (RED + saturation + deploys marked) and one runbook per alert.
3. Prove each alert fires (trigger on purpose in staging) and each probe reflects reality (kill a dependency, watch readiness drop).
4. Verify: incident drill passes using only these signals; no secret/PII in log samples.

## Failure modes

- Unstructured logs without IDs; metrics nobody alerts on; liveness==readiness single probe; alert fatigue from symptomless thresholds; traces unsampled at cost.

## Escalation

Vendor specifics via registry (opentelemetry, prometheus, grafana, sentry) + research/documentation-search; incident process itself is out of scope — hand to on-call with the runbook.

## References

- Registry: `../../research/documentation-search/references/framework-registry.yaml` (opentelemetry, prometheus, grafana, sentry).
- Related: `../../backend/production-readiness/`, `../../cloud/foundations/`.

From chnetajibc/skill-repo, MIT.

More from chnetajibc/skill-repo