querying-aws-cloudwatch

Files (1)

• SKILL.md

SKILL.md (first lines)

# Query AWS CloudWatch System Tables

## Overview

**Works best with** the [AWS MCP server](https://docs.aws.amazon.com/aws-mcp/) for sandboxed execution and audit logging. All commands below use the AWS CLI and work in any environment with configured AWS credentials.

The CloudWatch Logs S3 Tables integration exports log data as Apache Iceberg tables in the AWS-managed `aws-cloudwatch` table bucket. This enables SQL analysis via Amazon Athena and correlation of log data with non-CloudWatch data (S3 metadata, business tables, etc.). Available at no additional storage charge beyond CloudWatch ingestion pricing.

## Decision Tree

| User intent | Use this skill? | Alternative |
|---|---|---|
| Run SQL across large volumes of log data | **Yes** | — |
| Correlate logs with S3 metadata or other tables | **Yes** — join across catalogs | — |
| Quick log search / pattern matching | **No** | CloudWatch Logs Insights (faster for ad-hoc) |
| Real-time log streaming/tailing | **No** | CloudWatch Logs console or `logs filter-log-events` |
| Set up alarms on log patterns | **No** | CloudWatch Metric Filters / Alarms |
| Query historical logs before integration was enabled | **No** | CloudWatch Logs (no backfill in S3 Tables) |

## Supported Data Sources

The following data sources are available through the S3 Tables integration. Each data source has a namespace pattern used in SQL queries. Not all AWS vended data sources may be available in all Regions; check the CloudWatch console Data Sources tab for current availability.

| Data Source | Namespace pattern | Common use case |
|---|---|---|
| VPC Flow Logs | `amazon_vpc__flow` | Network traffic analysis, rejected connections |
| WAF Logs | `aws_waf__logs` | Blocked requests, rule hit analysis |
| CloudFront Access Logs | `amazon_cloudfront__access` | CDN traffic patterns, error rates |
| Route 53 Resolver Query Logs | `amazon_route53resolver__query` | DNS query analysis |
| Network Firewall Logs | `aws_networkfirewall__logs` | Firewall rule hits, dropped traffic |
| EKS Audit Logs | `amazon_eks__audit` | Kubernetes API audit trail |
| Verified Access Logs | `amazon_verifiedaccess__logs` | Zero-trust access decisions |
| SES Mail Logs | `amazon_ses__mail` | Email delivery/bounce tracking |
| VPC Lattice Access Logs | `amazon_vpclattice__access` | Service-to-service access patterns |
| Step Functions Logs | `aws_stepfunctions__logs` | Workflow execution debugging |
| Global Accelerator Flow Logs | `aws_globalaccelerator__flow` | Global network traffic |
| NLB Access Logs | `elastic_load_balancing__nlb_access` | Load balancer request tracing |
| Shield Logs | `aws_shield__logs` | DDoS mitigation events |
| Cognito Logs | `amazon_cognito__logs` | Auth/identity operations |
| ElastiCache Logs | `amazon_elasticache__logs` | Redis slow log, engine log |
| SageMaker Logs | `amazon_sagemaker__logs` | ML training/inference events |
| WorkMail Audit Logs | `amazon_workmail__audit` | Email security/compliance |
| Bedrock Agent Logs | `aws_bedrock_agent_core__logs` | AI agent invocations |
| Client VPN Logs | `aws_client_vpn__connections` | VPN connection tracking |
| Entity Resolution Logs | `aws_entity_resolution__logs` | Record matching operations |
| MediaPackage Access Logs | `aws_elemental_mediapackage__access` | Streaming delivery metrics |
| MediaTailor Logs | `aws_elemental_mediatailor__logs` | Ad insertion events |
| Transfer Family Logs | `aws_transfer_family__logs` | SFTP/FTPS file transfer tracking |
| Site-to-Site VPN Logs | `aws_site_to_site_vpn__logs` | VPN tunnel diagnostics |

> **Note**: This table lists the 24 most commonly queried data sources. The integration supports 43+ AWS vended data sources in total. Use `list-namespaces` on the `aws-cloudwatch` bucket to discover all available data sources in your account. Namespace patterns follow the convention `<service>__<type>`.

## Common Tasks

### 1. Check If Configured

```bash
# Check if the aws-cloudwatch table bucket exists
aws s3tables list-table-buckets --region <REGION> \
  --query "tableBuckets[?name=='aws-cloudwatch']"
```

- Empty result → integration not enabled. Guide user through setup.
- Bucket exists but no namespaces → integration enabled but no log data yet (only captures events *after* association).

List available tables:

```bash
aws s3tables list-namespaces --table-bucket-arn arn:aws:s3tables:<REGION>:<ACCOUNT>:bucket/aws-cloudwatch --region <REGION>

aws s3tables list-tables --table-bucket-arn arn:aws:s3tables:<REGION>:<ACCOUNT>:bucket/aws-cloudwatch --namespace <NAMESPACE> --region <REGION>
```

### 2. Enable / Configure

**Create integration:**

```bash
aws observabilityadmin create-s3-table-integration \
  --region <REGION> \
  --encryption '{"SseAlgorithm": "aws:kms", "KmsKeyArn": "<KMS_KEY_ARN>"}' \
  --role-arn <SERVICE_ROLE_ARN>
```

**Associate a specific data source (recommended):**

```bash
aws logs associate-source-to-s3-table-integration \
  --region <REGION> \
  --integration-arn <INTEGRATION_ARN> \
  --data-source '{"name": "<source-name>", "type": "<source-type>"}'
```

**Associate all data sources (wildcard):**

> ⚠️ **Warning**: Wildcard association delivers all current and future data sources to S3 Tables. Use specific associations for tighter control over what log data lands in queryable tables.

```bash
aws logs associate-source-to-s3-table-integration \
  --region <REGION> \
  --integration-arn <INTEGRATION_ARN> \
  --data-source '{"name": "*", "type": "*"}'
```

For IAM requirements (service role trust policy, permissions policy, condition keys), see [Security Considerations](#security-considerations) below.

### 3. Verify Permissions for Querying

Requires:

- S3 Tables federated catalog registered in Glue (`s3tablescatalog`)
- Lake Formation SELECT + DESCRIBE grants on the table (or IAM-only mode in supported regions)
- Athena execution permissions

Grant access:

```bash
aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=<ROLE_ARN> \

From aws/agent-toolkit-for-aws, Apache-2.0.

More from aws/agent-toolkit-for-aws