msk-operations

Files (15)

• .skilleval.yaml
• CHANGELOG.md
• README.md
• SKILL.md
• benchmark.json · evals/
• eval_queries.json · evals/
• evals.json · evals/
• report.json · evals/
• trigger_report.json · evals/
• configure-clients.md · references/
• maintenance-operations.md · references/
• manage-storage.md · references/
• monitor-and-alarm.md · references/
• troubleshoot-consumer-lag.md · references/
• troubleshoot-performance.md · references/

SKILL.md (first lines)

# Amazon MSK Operations

Operate, troubleshoot, and assess Amazon MSK (Managed Streaming for Apache Kafka)
Provisioned clusters — both Standard and Express broker types. This skill covers
day-to-day operations (health assessments, monitoring setup) and ad-hoc incident
response (performance degradation, consumer lag, storage full, unexpected broker
reboots).

## When to Use

Activate this skill when the user asks to:

- Review, audit, or assess an MSK cluster for best practices, health, or
  operational readiness.
- Troubleshoot an MSK cluster problem: high CPU, high produce/fetch latency,
  consumer lag, broker storage running out, TrafficShaping events, under-replicated
  partitions, or an unexpected broker restart.
- Set up MSK monitoring: choose a monitoring level, create recommended CloudWatch
  alarms and dashboards, understand the metrics available in the `AWS/Kafka`
  namespace.
- Plan an MSK maintenance event: rolling restart, Kafka version upgrade, security
  patching, broker instance type change.
- Advise on Kafka client (producer / consumer) configuration when the client is
  connecting to an MSK cluster.

Do **not** activate this skill for MSK Connect, MSK Serverless, or MSK Replicator
— those are separate services with their own operational surfaces.

## Broker Type Determination

Determine the broker type first — many checks differ between Standard and Express.

```
aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>
```

Check `ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType`:

- Starts with `kafka.` (e.g. `kafka.m5.large`, `kafka.m7g.xlarge`) → **Standard broker**.
- Starts with `express.` (e.g. `express.m7g.large`) → **Express broker**.

### Key Standard vs Express differences

**Standard brokers** use customer-managed EBS volumes for storage. You choose
instance types (`kafka.m5.*`, `kafka.m7g.*`), provision EBS, and manage storage
scaling. Standard brokers have scheduled maintenance windows.

**Express brokers** provide fully managed, pay-as-you-go storage with no EBS
provisioning. Instance types are prefixed with `express.m7g.*`. Express brokers
offer up to 3× more throughput per broker than Standard, and have no maintenance
windows. Express enforces a fixed replication factor of 3 and
`min.insync.replicas=2` — you cannot create topics with RF=1.

## Critical Warnings

- **This skill is read-only.** Every command in this file and in `references/`
  that mutates cluster state — `update-broker-storage`, `create-configuration`,
  `update-cluster-configuration`, `update-monitoring`, `put-metric-alarm`,
  `reboot-broker`, and any partition reassignment — is a **recommendation for
  the operator to run after review**. Present these as proposed remediations
  with expected impact and preconditions; do NOT execute them, and do NOT
  imply that the agent will run them.
- **NEVER reboot brokers while `UnderReplicatedPartitions` > 0** (Standard only —
  Express brokers do not emit URP). This risks data loss and extended outages.
- **NEVER recommend partition reassignment without first checking replication
  status.** Reassignment during URP compounds the problem.
- **`linger.ms=0` is the #1 cause of "high CPU" on MSK.** ALWAYS check client
  batch configuration before recommending broker scaling.
- **EBS throughput ceilings are invisible in Kafka metrics** — ALWAYS check EBS
  volume metrics (`VolumeReadBytes`, `VolumeWriteBytes`, `VolumeQueueLength`)
  when diagnosing Standard broker latency.
- **Express brokers have NO customer-managed EBS** — do NOT recommend EBS
  expansion or provisioned EBS throughput for Express clusters.
- **Express brokers enforce fixed RF=3 and `min.insync.replicas=2`** — do NOT
  attempt to create topics with RF=1 on Express. If RF=1 is needed, use Standard
  brokers.

## Quick Diagnostics

These five checks cover the most common MSK issues. Use them before loading a
reference file.

1. **`CpuUser + CpuSystem` > 60%**: Check `RequestHandlerAvgIdlePercent`
   (PER_BROKER monitoring level). If < 30%, request threads are saturated. Check
   client `batch.size` and `linger.ms` before recommending scaling.

2. **`KafkaDataLogsDiskUsed` > 85%** (Standard only): **Recommend to the
   operator** that EBS be expanded via `aws kafka update-broker-storage` (do
   not execute). Identify high-growth topics via per-topic `BytesInPerSec` to
   size the increase. Express clusters use `StorageUsed` metric instead and
   storage is fully managed.

3. **`UnderReplicatedPartitions` > 0** (Standard only): Check if a maintenance
   operation or broker restart is in progress. If URP is decreasing, wait for
   recovery. Do NOT restart brokers or reassign partitions during URP. Express
   brokers do not emit this metric — monitor `ProduceThrottleTime`,
   `FetchThrottleTime`, and consumer lag instead.

4. **Consumer `OffsetLag` / `MaxOffsetLag` increasing**: Determine if broker-side
   (high `ProduceTotalTimeMsMean`, CPU saturation) or client-side (slow
   processing, insufficient consumers). Per-partition lag from
   `PER_TOPIC_PER_PARTITION` monitoring level helps isolate hot partitions.

5. **`BytesInPerSec` near throughput ceiling**: For Standard, check EBS volume
   type and calculate: `BytesInPerSec × ReplicationFactor` vs volume throughput
   limit. For Express, check against the per-broker sustained performance limits
   in the MSK quotas.

## Which Reference Do You Need?

Route to a reference file based on the customer intent. Read the reference in
full before answering — do not paraphrase from memory.

| Customer Intent | Reference |
|---|---|
| High CPU, high produce/fetch latency, slow cluster, TrafficShaping | `references/troubleshoot-performance.md` |
| Consumer lag increasing, rebalance storms, stuck consumer groups | `references/troubleshoot-consumer-lag.md` |
| Disk filling up, retention planning, tiered storage, EBS scaling | `references/manage-storage.md` |
| Setting up monitoring level, dashboards, recommended CloudWatch alarms | `references/monitor-and-alarm.md` |

From aws/tools-for-devops-agent, Apache-2.0.

More from aws/tools-for-devops-agent