aws-observability

Files (50)

• SKILL.md
• alarm-template.ts · assets/cloudwatch/
• otel-config.yaml · assets/cloudwatch/
• agent-evaluation.md · references/cloudwatch-omni/
• alerts.md · references/cloudwatch-omni/
• concepts.md · references/cloudwatch-omni/
• context-graph.md · references/cloudwatch-omni/
• dashboards.md · references/cloudwatch-omni/
• programmatic-access.md · references/cloudwatch-omni/
• promql-metrics.md · references/cloudwatch-omni/query/
• sql-logs-traces.md · references/cloudwatch-omni/query/
• views.md · references/cloudwatch-omni/query/
• alarms.md · references/cloudwatch/
• cloudtrail.md · references/cloudwatch/
• dashboards.md · references/cloudwatch/
• dynamic-instrumentation.md · references/cloudwatch/
• breakpoint-creation.md · references/cloudwatch/dynamic-instrumentation/
• call-tree-and-directions.md · references/cloudwatch/dynamic-instrumentation/
• snapshot-parsing.md · references/cloudwatch/dynamic-instrumentation/
• log-insights.md · references/cloudwatch/
• metrics.md · references/cloudwatch/
• synthetics.md · references/cloudwatch/
• tracing.md · references/cloudwatch/
• troubleshooting.md · references/cloudwatch/
• capture_dataset_from_traces.py · scripts/cloudwatch-omni/
• evaluate_traces.py · scripts/cloudwatch-omni/
• test_evaluate_traces.py · scripts/cloudwatch-omni/
• di_app_signals_client.py · scripts/cloudwatch/
• di_capture.py · scripts/cloudwatch/
• di_constants.py · scripts/cloudwatch/
• di_crud_rendering.py · scripts/cloudwatch/
• di_crud_tools.py · scripts/cloudwatch/
• di_error_translation.py · scripts/cloudwatch/
• di_formatting.py · scripts/cloudwatch/
• di_gateway.py · scripts/cloudwatch/
• di_instrumentation.py · scripts/cloudwatch/
• di_location.py · scripts/cloudwatch/
• di_logs_client.py · scripts/cloudwatch/
• di_region.py · scripts/cloudwatch/
• di_result.py · scripts/cloudwatch/
• di_session.py · scripts/cloudwatch/
• di_snapshot_parsing.py · scripts/cloudwatch/
• di_snapshot_queries.py · scripts/cloudwatch/
• di_snapshot_rendering.py · scripts/cloudwatch/
• di_snapshot_tools.py · scripts/cloudwatch/
• di_snapshots.py · scripts/cloudwatch/
• di_status_assessment.py · scripts/cloudwatch/
• di_status_rendering.py · scripts/cloudwatch/
• di_status_tools.py · scripts/cloudwatch/
• di_validation.py · scripts/cloudwatch/

SKILL.md (first lines)

# AWS Observability

## Overview

Domain expertise for AWS observability across metrics, logs, and traces, for **two
products** that share the CloudWatch name but are separate services with separate
control planes, data models, and APIs:

| | CloudWatch | CloudWatch Omni |
|---|---|---|
| **What it is** | Log groups, metric namespaces, alarms, Log Insights, X-Ray, Application Signals | Application Observability / Agent Observability. A **Space** per account per Region is the access boundary over the account's CloudWatch **Dataset** (OpenTelemetry logs, traces, and metrics); the Dataset is a CloudWatch resource the Space reads, not something the Space contains |
| **Control plane** | `aws cloudwatch`, `aws logs`, `aws xray`, `aws application-signals` | `aws cloudwatchomni` (endpoint prefix `cloudwatch-omni`, signing name `cloudwatch`) |
| **Query** | Log Insights query language; GetMetricData | SQL over `logs.default` / `traces.default`; PromQL over metrics; named views |
| **Notify** | Alarms (metric, composite, anomaly) | **Alerts** (SQL/PromQL rule, contributors, OK/WARNING/CRITICAL/NODATA) |
| **Topology** | Application Signals service map | **Context graph** (GetContextGraph) |
| **Access** | IAM only | Domain → Space → **access grants** and **access profiles** (set up in `setting-up-cloudwatch-observability`) |
| **Only here** | Dynamic Instrumentation, Synthetics canaries, CloudTrail auditing, EMF | Agent-quality evaluation, views, context graph |
| **First-time setup** | Application Signals onboarding — ADOT auto-instrumentation, the `amazon-cloudwatch-observability` add-on, ServiceEvents, CI/CD metadata | Domain → Space → grants → telemetry in → plain-ADOT instrumentation |
| **References** | `references/cloudwatch/` | `references/cloudwatch-omni/` |

**This skill covers *using* both products, never setting either one up.** Both first-time
setup paths above live in **`setting-up-cloudwatch-observability`** (same two-folder split);
route there rather than describing an onboarding procedure from memory.

Enabling Omni does not replace CloudWatch; log groups, metrics, and alarms keep
working, and most customers use both.

**Works best with** the [AWS MCP server](https://docs.aws.amazon.com/aws-mcp/) — enables
running CLI commands, querying CloudWatch, and validating configurations directly. All
guidance also works with standard AWS CLI access.

**Note:** Reference files contain specific runtime versions, quota values, and feature
matrices that may change. When precision matters (e.g. deploying to production, choosing a
runtime, or checking a quota), confirm values against current AWS documentation rather
than relying solely on the values in these files.

## Scope guard — is this an observability request?

This skill owns the scope decision for every request routed to it, and this section is
its sole home.

**If a request is NOT about AWS observability with CloudWatch or CloudWatch Omni
(weather, trivia, general chit-chat, non-observability coding), decline it in one
sentence: state plainly that it is out of scope, never fabricate an answer, and never
claim a false capability limitation (no "no internet access", no "no weather data") —
the reason is simply that it is out of scope. Then redirect by naming what this skill
does cover: querying CloudWatch and CloudWatch Omni logs, traces, and metrics, building
dashboards, configuring alerts, investigating a service, or agent evaluation.**

Do **not** attempt the off-topic task, and do **not** call a tool or run a query in
pursuit of it. Keep it brief: no lecture, no long refusal.

## Step 0 — CloudWatch or CloudWatch Omni?

Decide this before routing. The natural wording ("set up an alert for high latency",
"build a dashboard", "query my logs") does not say which product the customer means.

1. **The customer names the product** — "Omni", "Application Observability", "Agent
   Observability", a Space, Domain, Dataset, access grant, `spaceId`, `cloudwatch-omni`,
   Omni SQL, PromQL, views, context graph, evaluators, agent evaluation / scoring traces /
   online or continuous evaluation / `gen_ai.evaluation`, and OTel span vocabulary
   (`traces.default`, `logs.default`, spans, `durationNano`, `status.code`, resource
   attributes, `service.name`) → **Omni**. "Log Insights", "log group", "metric
   namespace", "CloudWatch alarm", "metric alarm", "composite alarm", "anomaly alarm",
   "X-Ray", "Application Signals", "canary", "CloudTrail", "Dynamic Instrumentation" →
   **CloudWatch**. A "log group" named inside an agent-evaluation request is the
   online-evaluation data source to verify, not a CloudWatch signal. A bare "alarm" (or "alert") with no other product
   signal is ambiguous — fall through to rule 3 and probe: a Space → Omni alert
   ([alerts.md](references/cloudwatch-omni/alerts.md)); no Space → CloudWatch alarm
   ([cloudwatch/alarms.md](references/cloudwatch/alarms.md)). Exception: a "PromQL
   **alarm**" is a CloudWatch alarm on OTel metrics
   ([cloudwatch/alarms.md](references/cloudwatch/alarms.md)) — "alarm" wins over "PromQL".
2. **Knowledge or how-to question** ("what is an Omni alert", "does Omni have an API",
   "how do alerts differ from alarms") → answer from the reference files directly. Do
   **not** probe the account, and do not divert to the other product. Whether a
   capability exists is a fact about the product, not the account.
   [concepts.md](references/cloudwatch-omni/concepts.md) carries the full feature-equivalence matrix.
2a. **Authoring / how-to alert request** — "create / set up / write an Omni alert that
   fires when X", "how should I alert on Y" — with no Space or Region supplied and no
   go-ahead to actually create it, is a **HOW-TO** request. Deliver the authoring guidance
   (the alert row of "Must-state checklists" below, then
   [alerts.md](references/cloudwatch-omni/alerts.md)) **FIRST**, from the reference files. Do
   **NOT** stall on a Region/Space clarifying question and do **NOT** fall back to a
   CloudWatch alarm for a request that explicitly says "Omni alert". Probe the account only
   once the user supplies a Space/Region or asks you to create it.
2b. **A telemetry object is already in the prompt** — the user pastes, or the UI passes as
   context, a span, trace, log record, or query result and asks what it means, what errors
   it has, or how long it took. This is **not** a live-data request: do not probe the
   account, run a query, or ask the user to fetch it again. Read it in place per the
   "Reading a span or trace you already have" section of
   [query/sql-logs-traces.md](references/cloudwatch-omni/query/sql-logs-traces.md) and
   state every item in its must-state list.
2c. **A query request whose live result is empty or unreachable** ("show me the slowest
   spans", "which traces failed") is still answered with the **methodology** — the exact
   query and the field rules that make it correct (see the must-state callouts in
   [query/sql-logs-traces.md](references/cloudwatch-omni/query/sql-logs-traces.md)). Zero
   rows, a wrong-Region Space, or an unreachable endpoint is reported as a finding
   alongside the query, never as the whole answer.
3. **Request that must act on live data, and the wording is ambiguous** (not a Step 0.5
   content question, which is answered from the catalog) → probe the target Region first:

   ```
   aws cloudwatchomni list-domains
   aws cloudwatchomni list-spaces --region <target-region> --query "items[?region=='<target-region>']"  # account-global; filter to the Region
   ```

   - `list-spaces` is **account-global** (the `--region` flag only selects the endpoint), so filter its result to the target Region as shown rather than reading a non-empty list as proof; a Space is one per account per Region. See `references/cloudwatch-omni/concepts.md` for the full Region-probe rationale.
   - A Domain and a Space in that Region (a non-empty filtered list) → **Omni**.
   - No Space in that Region (an empty filtered list) → **CloudWatch**: alarms → [cloudwatch/alarms.md](references/cloudwatch/alarms.md),
     dashboards → [cloudwatch/dashboards.md](references/cloudwatch/dashboards.md), queries →
     [cloudwatch/log-insights.md](references/cloudwatch/log-insights.md), metrics →
     [cloudwatch/metrics.md](references/cloudwatch/metrics.md). If the customer explicitly asked
     for Omni and has no Space, that is a first-time setup — see step 4.
   - The probe itself errors ("not yet supported", unknown service, endpoint does not
     resolve) → the CLI/SDK model in use lacks `cloudwatchomni`. That is **not** evidence
     Omni is absent and must not be reported as "Omni is unavailable". The customer's
     installed AWS CLI/SDK most likely predates the service. If the request carried
     **any** Omni signal, give the customer the upgrade command to run (AWS CLI v2
     reinstall or `brew upgrade awscli`; `pip install -U boto3 botocore`) and have them

From aws/agent-toolkit-for-aws, Apache-2.0.

More from aws/agent-toolkit-for-aws