bedrock-operation-review

Files (12)

• .skilleval.yaml
• CHANGELOG.md
• README.md
• SKILL.md
• benchmark.json · evals/
• eval_queries.json · evals/
• evals.json · evals/
• bedrock-context.json · evals/files/
• report.json · evals/
• trigger_report.json · evals/
• best-practices-checklist.md · references/
• metrics-thresholds.md · references/

SKILL.md (first lines)

# Amazon Bedrock Operational Review

Conduct a comprehensive operational review of Amazon Bedrock workloads aligned with the
[AWS Well-Architected Framework](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html)
and [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html)
best practices.

This skill uses the **AWS Bedrock, Bedrock Agent, CloudWatch, Service Quotas, and
EC2 APIs only** — all data is collected through native AWS control-plane APIs and
CloudWatch metrics. It performs no data-plane model invocations and reads no prompt
or response content.

## When to Use

Activate this skill when the user asks to:
- Review, audit, or assess an Amazon Bedrock workload
- Check Bedrock best-practices compliance
- Evaluate Bedrock security, performance, cost, quotas, or resilience
- Perform a Bedrock operational readiness review (ORR)
- Investigate Bedrock configuration drift, throttling, or cost drivers

## Step 1: Identify Target Scope

Ask the user which accounts and regions to review. Accept:
- Specific account IDs and regions
- "all regions" for a given account
- A specific pillar or set of checks (e.g. "just cost optimization")

If no scope is given, default to all configured account regions and all pillars.
Default the analysis window to the last 7 days unless the user specifies a range
(historical windows older than ~2 weeks may have reduced CloudWatch resolution).

## Step 2: Discover Bedrock Resources

Per account/region, begin by checking whether the account actually uses Bedrock in
this region. **Do not use `bedrock.ListFoundationModels` as an activity signal** — it
returns the regional model catalog available to the account and is generally non-empty
in every supported region regardless of usage, so it is treated as catalog data only
(see [ListFoundationModels](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListFoundationModels.html)).

Instead, determine activity from **account-owned resources and `AWS/Bedrock` metrics**:
- Account-owned resources: `bedrockagent.ListAgents`, `bedrockagent.ListKnowledgeBases`,
  `bedrock.ListGuardrails`, `bedrock.ListCustomModels`,
  `bedrock.ListProvisionedModelThroughputs`, `bedrock.ListInferenceProfiles`
  (application-scoped), and `bedrock.ListModelCustomizationJobs`.
- Invocation activity: `cloudwatch.ListMetrics` for the `AWS/Bedrock` namespace.

**If no account-owned Bedrock resources exist AND CloudWatch shows no `AWS/Bedrock`
metrics for the region, record "No Bedrock activity detected — skipping pillar
analysis" for that region and move on. Do not generate empty findings tables for
inactive regions.**

For regions where Bedrock is active, collect the full resource inventory:

```
bedrock.ListFoundationModels                   # catalog reference only: model
                                               # availability + lifecycle status
                                               # (NOT an account-activity signal)
bedrock.ListGuardrails / GetGuardrail          # configured guardrails
bedrock.GetModelInvocationLoggingConfiguration
bedrock.ListInferenceProfiles / GetInferenceProfile
bedrock.ListPromptRouters / GetPromptRouter
bedrock.ListProvisionedModelThroughputs / GetProvisionedModelThroughput
bedrock.ListCustomModels / GetCustomModel
bedrock.ListModelCustomizationJobs / GetModelCustomizationJob
bedrockagent.ListAgents / GetAgent             # agents (draft version)
bedrockagent.ListAgentVersions / GetAgentVersion  # deployed versions — read only the
                                               # per-version model ID, orchestration
                                               # type, and whether a prompt override is
                                               # present. Do NOT read the override
                                               # base-prompt template text.
bedrockagent.ListAgentAliases
bedrockagent.ListKnowledgeBases / GetKnowledgeBase
bedrockagent.ListDataSources / GetDataSource
bedrockagent.ListPrompts                       # Prompt Management: presence/metadata
                                               # only (identifiers, versions, counts).
                                               # Do NOT call GetPrompt — it returns
                                               # prompt template/variant content, which
                                               # this skill does not read.
```

Capture per resource: identifiers, ARNs, status, creation/update timestamps,
encryption configuration (AWS-managed vs customer-managed KMS key), and any
VPC configuration.

## Step 3: Collect CloudWatch Metrics (namespace `AWS/Bedrock`)

**Before querying metrics, load the authoritative thresholds reference:**
```
read_skill_resource(skill_id="bedrock-operation-review", path="references/metrics-thresholds.md")
```
Use the thresholds from that file when classifying metric values as Normal, Warning,
or Critical throughout this step and Step 4.

Discover which models are actually invoked with `cloudwatch.ListMetrics`
(dimension `ModelId`), then pull metric data with `cloudwatch.GetMetricData`.
Use the user's selected window; default to 7 days.

Key model-level metrics (dimension `ModelId`):

| Metric | Stat | Signal |
|--------|------|--------|
| Invocations | Sum | Volume / denominator for rate calcs |
| InvocationThrottles | Sum | Throttling / quota pressure |
| InvocationServerErrors | Sum | Server-side failures |
| InvocationClientErrors | Sum | Bad input / blocked content |
| InvocationLatency | Average, p95 | End-to-end latency |
| TimeToFirstToken | Average, p95 | Perceived latency (streaming) |
| InputTokenCount | Sum | Input token volume |
| OutputTokenCount | Sum | Output token volume |
| InvocationsIntervened | Sum | Guardrail interventions |
| TextUnitCount | Sum | Guardrail text-unit consumption |
| CacheReadInputTokenCount | Sum | Prompt cache reads |
| CacheWriteInputTokenCount | Sum | Prompt cache writes |

GPU metrics for self-managed workloads live in the `CWAgent` namespace
(`nvidia_smi_utilization_gpu`, `nvidia_smi_memory_util`).

`cloudwatch.GetMetricData` allows up to 500 metric-data queries per call — batch

From aws/tools-for-devops-agent, Apache-2.0.

More from aws/tools-for-devops-agent