managing-amazon-msk

Files (16)

• SKILL.md
• configure-clients.md · references/
• configure-cluster.md · references/
• data-delivery-for-general-purpose-s3.md · references/
• maintenance-operations.md · references/
• manage-storage.md · references/
• manage-topics.md · references/
• monitor-and-alarm.md · references/
• msk-replicator.md · references/
• size-and-choose-cluster.md · references/
• streaming-tables-troubleshooting.md · references/
• streaming-tables.md · references/
• troubleshoot-consumer-lag.md · references/
• troubleshoot-performance.md · references/
• zk-to-kraft-migration.md · references/
• msk_sizing.py · scripts/

SKILL.md (first lines)

# Amazon MSK

## Overview

Domain expertise for operating Amazon MSK Provisioned clusters with Standard and Express broker types. Covers performance troubleshooting, consumer lag diagnosis, storage management, cluster sizing, client configuration, and CloudWatch monitoring.

Execute commands using available tools from the AWS MCP server when connected — it provides sandboxed execution, audit logging, and observability. When the MCP server is not available, fall back to the AWS CLI or shell as needed.

**Standard brokers** use customer-managed EBS volumes for storage. You choose instance types (kafka.m5/m7g families), provision EBS, and manage storage scaling.

**Express brokers** use instance types prefixed with `express.m7g` and are the default recommendation for almost all MSK workloads — they typically cost less, not just less effort. Up to 3x ingress per broker ([MSK Express broker types](https://docs.aws.amazon.com/msk/latest/developerguide/msk-broker-types-express.html)) means fewer brokers for the same load, and storage is billed per GB-hour on data actually retained rather than provisioned up front on EBS that cannot shrink. They also scale 20x faster, rebalance partitions 180x faster ([Intelligent Rebalancing](https://docs.aws.amazon.com/msk/latest/developerguide/intelligent-rebalancing-self-balancing-paritions.html)), recover 90% quicker ([MSK Express broker types](https://docs.aws.amazon.com/msk/latest/developerguide/msk-broker-types-express.html)), and have no maintenance windows. Express brokers have NO customer-managed EBS — do NOT recommend EBS expansion or provisioned throughput for Express clusters. Express brokers enforce fixed replication factor of 3 and `min.insync.replicas=2`. See [size-and-choose-cluster.md](references/size-and-choose-cluster.md) for the full Standard vs Express decision framework.

## Which Workflow Do You Need?

Determine the broker type first: `aws kafka describe-cluster-v2 --cluster-arn <arn>`. Check `Provisioned.BrokerNodeGroupInfo.InstanceType` — if it starts with `express.`, it is an Express cluster.

| Customer Intent | Reference |
|---|---|
| High CPU, high latency, slow cluster, traffic shaping | [troubleshoot-performance.md](references/troubleshoot-performance.md) |
| Consumer lag increasing, rebalance storms, stuck consumer groups | [troubleshoot-consumer-lag.md](references/troubleshoot-consumer-lag.md) |
| Disk filling up, retention planning, tiered storage | [manage-storage.md](references/manage-storage.md) |
| Choosing Standard vs Express, sizing a cluster, partition limits, broker count, monthly cost | [size-and-choose-cluster.md](references/size-and-choose-cluster.md) |
| Producer/consumer configuration, IAM/SCRAM/TLS auth | [configure-clients.md](references/configure-clients.md) |
| Creating/applying MSK configurations (`server.properties`); custom domain names on brokers via `custom.advertised.listeners` (advertised listeners, static/custom bootstrap endpoint) — validation rules, apply/rollback, scaling; migrating from the dynamic per-broker `kafka-configs.sh` override to the static property | [configure-cluster.md](references/configure-cluster.md) |
| Client-side connectivity for a custom domain: NLB + ACM certificate + Route 53 fronting, TLS handshake/termination, mTLS through an NLB, cross-zone load balancing | [configure-clients.md](references/configure-clients.md) (Custom Domain Name Connectivity section) |
| Create, update, delete, list, or describe Kafka topics; when to use the native topic APIs / `CfnTopic` vs. a Kafka admin client | [manage-topics.md](references/manage-topics.md) |
| MSK Replicator setup, migration, DR, active-active/active-passive, cross-account replication, self-managed-to-MSK or MSK-to-MSK, loop-prevention/topic-naming behavior | [msk-replicator.md](references/msk-replicator.md) |
| Replication throughput per partition, how many partitions a replicator needs, why replicator scaling isn't increasing throughput, replicator quotas (topics, ingress, record size), `ReplicationLatency`/`MessageLag`/`ReplicatorThroughput` | [msk-replicator.md](references/msk-replicator.md) (Size partition count section) |
| Setting up monitoring, dashboards, alarms | [monitor-and-alarm.md](references/monitor-and-alarm.md) |
| Full CloudWatch metric list (Standard or Express) | Prefer [monitor-and-alarm.md](references/monitor-and-alarm.md) for strategic recommendations and how to interpret metrics, only search documentation if you need to understand a metric not included in this reference file ([MSK Standard CloudWatch Metrics](https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html), [MSK Express CloudWatch Metrics](https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details-express.html)) for full list |
| Rolling restart impact, patching, maintenance resilience | [maintenance-operations.md](references/maintenance-operations.md) |
| Deliver streaming data to Apache Iceberg tables on S3 Tables with low cost in a fully managed service (Streaming Tables) — setup, IAM, schema, create/update/delete/list/describe channels | [streaming-tables.md](references/streaming-tables.md) |
| Deliver topic data to S3 bucket as JSON/ByteArray/String objects with low cost in a fully managed service (Data Delivery for General Purpose S3 buckets) — setup, IAM, output key templates, create/update/delete/list/describe channels | [data-delivery-for-general-purpose-s3.md](references/data-delivery-for-general-purpose-s3.md) |
| Build a lakehouse / data lake from Kafka; make streaming data queryable in Athena | [streaming-tables.md](references/streaming-tables.md) |
| Alternative to Kafka Connect S3 Sink or Amazon Data Firehose for MSK; zero-ops streaming delivery to S3 | [data-delivery-for-general-purpose-s3.md](references/data-delivery-for-general-purpose-s3.md) |
| Streaming Tables / Data Delivery CloudWatch metrics and alarms, DLQ errors, failed deliveries, channel state transitions, freshness lag | [streaming-tables-troubleshooting.md](references/streaming-tables-troubleshooting.md) |
| "Can I use Streaming Tables / Data Delivery on MSK Serverless / Standard brokers?" — eligibility routing | [streaming-tables.md](references/streaming-tables.md) (answer is always: Express brokers only, use Firehose, Flink, or Kafka Connect for Standard and Serqverless - [Firehose integration for Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/integrations-kinesis-data-firehose.html)) |
| What are the current supported Kafka versions for MSK? | [Supported Apache Kafka versions](https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html) |
| Does MSK support KRaft clusters, and how do I migrate between ZooKeeper and KRaft mode clusters? | [zk-to-kraft-migration.md](references/zk-to-kraft-migration.md) |
| What are the current quotas for MSK Express (ingress, egress, partitions, broker count, etc.)? | [MSK Express Quotas](https://docs.aws.amazon.com/msk/latest/developerguide/limits.html#msk-express-quota) |
| What are the current quotas for MSK Standard (partitions, broker count, etc.)? | [MSK Standard Quotas](https://docs.aws.amazon.com/msk/latest/developerguide/limits.html#msk-provisioned-quota), and [MSK Standard best practices](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html#standard-server-side-considerations) for partition count limits |
| What broker-level configuration changes can I make on MSK Express or Standard brokers? | [MSK Configuration](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration.html) |

## Available scripts

- **`scripts/msk_sizing.py`** — **MUST** be run for any sizing question (broker count, instance choice, cost). Prices from the live AWS Pricing API when AWS credentials with `pricing:GetProducts` are available, and falls back to an embedded `us-east-1` snapshot when they are not, so it runs with or without credentials. See [size-and-choose-cluster.md](references/size-and-choose-cluster.md) for the required workflow and script reference.

## Guardrail — where this skill's own files live (MCP vs local install)

This skill can be loaded two ways, and they resolve the skill's **own bundled files** — the `references/` documents and the `scripts/` files
from different places. Determine how the skill was loaded before you read a reference or run a script:

- **Loaded through the AWS MCP `retrieve_skill` tool call.** The skill is **not
  installed on the local filesystem**; its reference files and scripts do not
  exist on disk. You MUST fetch each reference or script through the same
  `retrieve_skill` tool by passing the `file` parameter (for example,
  `file="references/configure-clients.md"` or `file="scripts/msk_sizing.py"`),
  and run a script from the content that tool returns. Do NOT `file_read` these
  paths from the local or working directory, and do NOT search the filesystem
  for them — they are not there, and any local file that happens to match the
  name is unrelated to this skill.
- **Installed locally** (the skill lives in a local skills directory such as
  `.claude/skills/managing-amazon-msk/`, `~/.claude/skills/managing-amazon-msk/`, or
  `.kiro/skills/managing-amazon-msk/`). Read references and run scripts from the
  local skill directory using the relative paths shown throughout this
  documentation.

This distinction applies **only** to the skill's own packaged files. Every artifact
created during a session or supplied by users are read from and written to
the user's working directory regardless of how the skill was loaded. Never
fetch or write customer data through `retrieve_skill`.

## Common Workflows

**Create/apply Amazon MSK configurations and set custom domain names** — creating an Amazon MSK configuration (`server.properties` with the `fileb://` real-newline requirement), applying it with `update-cluster-configuration`, and setting broker custom domain names via `custom.advertised.listeners`: see [configure-cluster.md](references/configure-cluster.md). For the NLB/certificate/DNS connectivity that fronts a custom domain, see [configure-clients.md](references/configure-clients.md).

From aws/agent-toolkit-for-aws, Apache-2.0.

More from aws/agent-toolkit-for-aws