agents-optimize

Files (4)

• SKILL.md
• cost.md · references/
• evals.md · references/
• observability.md · references/

SKILL.md (first lines)

# optimize

Measure and improve your AgentCore agent's quality through evaluation, monitoring, and observability.

## When to use

- You want to know if your agent is giving good answers
- You want to set up continuous quality monitoring in production
- You want to add a quality gate to your CI/CD pipeline
- You want to understand agent behavior through logs, metrics, and traces
- You want to set up CloudWatch dashboards or X-Ray tracing

Do NOT use for:

- Debugging a specific broken agent (wrong answers, errors) → use `agents-debug`
- Production security hardening (IAM, auth) → use `agents-harden`

## Input

`$ARGUMENTS` can be:

- An eval goal: "add a quality gate", "set up monitoring"
- An observability goal: "set up CloudWatch dashboard", "understand my traces"
- A specific evaluator: "llm-as-a-judge", "code-based"
- Empty — the skill will guide based on project context

## Process

### Step 0: Verify CLI version

Run `agentcore --version`. This skill requires v0.9.0 or later.

### Step 1: Read project context

Read `agentcore/agentcore.json` to understand existing evaluators, online eval configs, and agent setup.

If `agentcore/agentcore.json` is not found:
> "This skill requires an AgentCore project. Use `agents-get-started` to create one."

### Step 2: Determine the workflow

| Developer intent | Action |
|---|---|
| Measure quality, add evaluator, run eval, CI/CD gate, online monitoring | Load [`references/evals.md`](references/evals.md) and follow its workflow |
| Set up observability, CloudWatch, X-Ray, logs, metrics, dashboards | Load [`references/observability.md`](references/observability.md) and follow its workflow |
| Understand or reduce AgentCore costs | Load [`references/cost.md`](references/cost.md) |
| Both — "I want to understand and improve my agent" | Start with observability setup, then add evals |

### Step 3: Follow the loaded reference

The reference file contains the full procedure. Follow it step by step.

### Cross-references

- After setting up evals, suggest `agents-harden` for production readiness
- If eval results reveal agent issues, suggest `agents-debug` for root cause analysis
- If the developer needs to add capabilities first, suggest `agents-build`

## Output

Depends on the workflow — see the loaded reference for specific outputs.

## Quality criteria

- Evaluator configuration uses only valid CLI flags
- Online eval sampling rate is appropriate (not 100% in production without discussion)
- CI/CD quality gate has a clear pass/fail threshold
- Observability setup includes both tracing and logging
- The developer understands the eval data delay: **~10 seconds put-to-get, end-to-end** — one ingestion step covers both trace reads and eval queries; there is no separate indexing wait

From aws/agent-toolkit-for-aws, Apache-2.0.

More from aws/agent-toolkit-for-aws