investigating-incidents-with-aws-devops-agent

Files (2)

• REFERENCE.md
• SKILL.md

SKILL.md (first lines)

# Investigate an AWS incident

> **AgentSpace routing (SigV4 only):** If `list_agent_spaces` is available in your tool list and the multi-space orchestration skill has NOT been invoked yet this session, invoke it first to determine which `agent_space_id` to use. Then pass `agent_space_id` on all tool calls below. For bearer token auth this is unnecessary — the token is already scoped to one space.

Use this when the user is reporting or describing an operational problem that needs deep async analysis (5–8 minutes of agent work). For fast questions about cost, architecture, or topology, use the `chatting-with-aws-devops-agent` skill instead.

## Pre-flight

Before starting an investigation, gather **local context** and pack it into the `title` parameter. This is the killer feature — the DevOps Agent knows your AWS cloud; you know the user's local workspace.

Always collect:

- Service identity from `package.json` / `pom.xml` / `Cargo.toml` / `requirements.txt` / `Makefile`
- `git log --oneline -10` (recent commits — agent correlates deploys to incidents)
- `git diff --stat` (uncommitted work that might be relevant)

When investigating errors, also include:

- The full stack trace or relevant log excerpt
- Any IaC files relevant to the failing resource (CDK / CloudFormation / Terraform / ECS task def)

## Start the investigation

```
aws_devops_agent__investigate(
    title="ECS 503 errors on checkout-service since commit abc1234 deployed 2h ago. CDK: ECS Fargate behind ALB. Error: upstream connect error."
)
→ {"status": "investigation_started", "taskId": "...", "executionId": "...", "message": "...", "next_steps": "..."}
```

Save the `taskId` and `executionId`.

> **Tip:** Pack as much context as possible into the `title` — service name, error type, time window, recent deploys. The agent uses this to scope its analysis.

## Stream progress — never silently poll

**Investigations take 5–8 minutes. Tell the user up front, then keep them informed.**

Loop every 30–45 seconds:

### 1. Check status

```
aws_devops_agent__get_task(task_id="TASK_ID")
→ {"task": {"taskId": "...", "status": "IN_PROGRESS", ...}}
```

### 2. Fetch new findings

```
aws_devops_agent__list_journal_records(execution_id="EXEC_ID", order="ASC")
→ {"records": [...]}
```

Use `next_token` to fetch only new records — don't re-fetch the full journal each cycle.

### 3. Summarize progress to the user

Map record types to emoji prefixes:

- `PLANNING` → 📋 planning approach
- `SEARCHING` → 🔍 querying CloudWatch / X-Ray / logs
- `ANALYSIS` → 🔬 analyzing
- `FINDING` → 🎯 key discovery (highlight this)
- `ACTION` → 🔧 taking an action
- `SUMMARY` → 📊 final summary
- `SUGGESTION` → 💡 recommended fix

Example updates:
> 🔬 **2 min in:** Agent found error rate spiked to 23% at 14:32 UTC. Checking X-Ray traces for downstream failures.
>
> 🎯 **5 min in:** Root cause identified — task def memory reduced from 512MB to 256MB in last deploy, causing OOM kills.

## On COMPLETED

### 1. Get final findings

```
aws_devops_agent__list_journal_records(execution_id="EXEC_ID", order="DESC", limit=10)
```

### 2. Get recommendations

```
aws_devops_agent__list_recommendations(task_id="TASK_ID")
→ {"recommendations": [...]}
```

For detailed mitigation specs:

```
aws_devops_agent__get_recommendation(recommendation_id="REC_ID")
```

### 3. Present to the user

If recommendations contain IaC changes (CDK / CFN / Terraform), generate the fix locally **but do not apply it**. Show the diff, explain it, and let the user approve.

## Fallback path (aws-mcp)

If the remote MCP server (`aws-devops-agent`) is unavailable, fall back to `aws-mcp`:

```
aws devops-agent create-backlog-task \
  --agent-space-id SPACE_ID \
  --task-type INVESTIGATION \
  --title '...' \
  --priority HIGH \
  --description '...' \
  --region us-east-1
→ taskId
```

Then poll with:

```
aws devops-agent get-backlog-task --agent-space-id SPACE_ID --task-id TASK_ID --region us-east-1
```

From aws/agent-toolkit-for-aws, Apache-2.0.

More from aws/agent-toolkit-for-aws