sagemaker-ai-ops-review

Files (9)

• .skilleval.yaml
• CHANGELOG.md
• README.md
• SKILL.md
• eval_queries.json · evals/
• evals.json · evals/
• exemptions.json · evals/
• iam-policy.json · references/
• pillar-checks.md · references/

SKILL.md (first lines)

# Amazon SageMaker AI Operational Review

Run the Amazon SageMaker AI operational review checks against a customer's Amazon SageMaker AI
resources and produce an **Amazon SageMaker AI Operational Review** report. It evaluates
**8 pillars, 20 checks** using native AWS APIs (via `use_aws`). The Best Practices pillar's
recommendations are grounded in the public AWS Well-Architected lenses — see `pillar-checks.md`.

This is a strict **READ-ONLY** review: data is collected through native AWS `List*` /
`Describe*` control-plane APIs, CloudWatch metric reads, `servicequotas:GetServiceQuota`,
`health:DescribeEvents`, and `savingsplans:DescribeSavingsPlans`. It performs no model
invocations, launches no jobs, and reads no inference payloads.

## When to Use

Activate this skill when the user asks to review, audit, or assess an Amazon SageMaker AI
workload, check SageMaker AI best-practices posture, or run an **Operational Readiness Review
(ORR)** for SageMaker AI — for one pillar, a subset of checks, or the full set.

The **ORR** use case is the primary one: run this review **before a team deploys a SageMaker AI
workload to production**, as the readiness gate. Because every finding is severity-ranked and
carries a concrete remediation, the report doubles as the pre-production punch list — clear the
High and Medium findings, then launch. It is equally suited to a recurring cadence afterwards
(weekly or monthly posture review) and to an ad-hoc audit of a newly inherited account.

## Pillars and Checks

Run checks grouped by pillar in the order below. **Load `references/pillar-checks.md`** for
each check's APIs, logic, thresholds, and output fields.

| Pillar | Checks |
|--------|--------|
| **Security** | Check Encryption · SageMaker VPC Check · VPC Configuration Check |
| **Performance** | SageMaker Endpoint Inference Type · SageMaker Endpoint Latency |
| **Cost Optimization** | SageMaker Resource Tagging Check · Trainium and Inferentia Usage · Autoscaling Endpoint Check · Sagemaker Savings Plan · Sagemaker Lifecycle Configurations · Sagemaker Inference Recommender Jobs Check · Sagemaker Stale Endpoints Check |
| **Service Quotas** | Service Quotas Check |
| **Resiliency** | SageMaker Endpoint Instances · SageMaker Lifecycle Events |
| **Operational Excellence** | Sagemaker Project Check · Sagemaker Pipeline Check · SageMaker Endpoint Datacapture Enabled Check |
| **Sustainability** | Domain Region Check |
| **Best Practices** | Well-Architected Recommendations (SageMaker AI) |

## Step 1: Identify Scope

Confirm with the user:
- **Account IDs** and **regions** to review (default: current account via `sts:GetCallerIdentity`). If regions are unspecified, discover active regions with `ce:GetCostAndUsage` (SERVICE = "Amazon SageMaker", grouped by REGION); Cost Explorer is payer-scoped, so if it returns nothing, **fall back** to sweeping a default region set with `sagemaker.list-endpoints`/`list-domains`/`list-notebook-instances`. Conclude "no activity" only after both come back empty.
- **Pillars or individual checks** to run (default: all 8 pillars / 20 checks).
- **Date range** for time-windowed checks (Latency = last 7 days, Stale Endpoints = last 90 days, Service Quotas usage = **trailing 24 hours** — these windows are fixed by the checks and must not be shortened; `references/pillar-checks.md` is authoritative on each).

## Step 2: Run the Checks

For each in-scope check, call the APIs listed in `references/pillar-checks.md` via `use_aws`
and build the check's result rows. Follow this behavior:

- **Read-only.** `List*` then `Describe*`; paginate every call that returns a token.
- **Per-check isolation.** Catch and record errors per check as a `{ error }` row — a failed
  check never aborts the review.
- **Three APIs are global — call them once in `us-east-1`, never inside the per-region loop:**
  `health` (`describe-events`, `describe-affected-entities`), `ce` (`get-cost-and-usage`), and
  `savingsplans` (`describe-savings-plans`). They have no regional endpoints. Looping them per
  region fails everywhere but `us-east-1`, and the failure mimics the checks' legitimate
  degradation paths — a Health error looks like "no Business/Enterprise Support plan", a Savings
  Plans error looks like "permission not granted" — so the report states a plausible wrong reason
  instead of surfacing a bug. Health returns events for all regions; filter to the in-scope
  regions client-side.
- **Units are part of every number.** Where a metric has a unit, the report carries it. In
  particular `ModelLatency` / `OverheadLatency` are published in **microseconds** — label the
  column and also give the millisecond conversion. An unlabelled six-figure latency reads as
  milliseconds and manufactures a false performance escalation.
- **Permissions / graceful degradation.** Nearly all APIs are covered by the AWS-managed
  `AIDevOpsAgentAccessPolicy` on the DevOps Agent role. The one exception —
  `savingsplans:DescribeSavingsPlans` (Savings Plan check) — is an optional add-on. The AWS
  Health APIs used by the Lifecycle Events check are covered by the managed policy but
  additionally require a Business/Enterprise Support plan. On AccessDenied for a check, report it
  as **"not evaluated — permission not granted"** and continue; never emit a false "none found"
  from an access error.
- **Empty results** (permission present, nothing there) produce a single "No <resource> found"
  row, not a dropped section.
- **Severity-ranked findings.** Assign each finding a severity per `references/pillar-checks.md`:
  **High**, **Medium**, **Low**, or **Informational** (inventory checks with no pass/fail signal).
  Checks with a compliance signal set severity as defined there — e.g. Studio domain not `VpcOnly`
  → High; no autoscaling / an Inference Component endpoint whose host instance fleet is fixed while its
  components autoscale / idle endpoint at least 90 days old / notebook with no customer-managed
  KMS key / no VPC config / Savings Plan expired or within 30 days of expiry / AWS Health event with
  `actionability = ACTION_REQUIRED` → Medium; missing tags / data capture disabled → Low. The Service
  Quotas Check derives its tier from utilization (≥ 90% High, ≥ 75% Medium, else Low; Unknown if no
  usage data).
- **One finding = one non-compliant resource in one check**, keyed by `(check, region, resource)`.
  Do **not** aggregate resources into a single finding — three notebooks with no customer-managed
  key are three Medium findings, not one. Aggregation breaks the severity counts and makes runs incomparable.
- **One recommendation per High or Medium finding.** Emit exactly one concrete, SageMaker-specific
  recommendation for every High and Medium finding. Low and Informational findings do not require one.
- Use only the severities each check defines; do **not** invent thresholds a check does not define.

## Step 3: Generate the Report

Produce a single Markdown report titled **"Amazon SageMaker AI Operational Review"**, with the
structure below.

```markdown
# Amazon SageMaker AI Operational Review

**Account IDs:** <comma-separated account IDs>
**Regions:** <comma-separated regions>
**Date Range:** <range or "Not specified">

> **AI Disclaimer:** The AI-generated insights in this report are provided for informational purposes only. They should be reviewed and validated by qualified personnel before taking any action. AWS is not responsible for any decisions made based on AI-generated content.

## Executive Summary

<severity-ranked roll-up of findings across all pillars: count by severity (High / Medium /
Low), then the High and Medium findings listed most-severe first, each with its one-line
recommendation. Omit only if there are no High/Medium/Low findings at all.>

## <Pillar Name>

### <Check Name>

**Guidance**

<what the check evaluates and the relevant SageMaker best practice>

From aws/tools-for-devops-agent, Apache-2.0.

More from aws/tools-for-devops-agent