aiml-access-diagnostics

Files (12)

• .skilleval.yaml
• CHANGELOG.md
• README.md
• SKILL.md
• eval_queries.json · evals/
• evals.json · evals/
• access-chain-model.md · references/
• data-collection.md · references/
• finding-logic.md · references/
• report-format.md · references/
• svc-bedrock.md · references/
• svc-sagemaker.md · references/

SKILL.md (first lines)

# AI/ML Access Diagnostics

Diagnose why an AI/ML service call was denied. Walk the authorization chain hop by
hop, name the hop that denied the call, and propose a scoped IAM policy for human
review. Read-only throughout.

## Checklist

Work through these steps in order. Each is detailed in its own section below.

- [ ] **Step 1 — Classify the request:** confirm the service is Bedrock or SageMaker, and that there is an observed failure (not a speculative audit). Stop otherwise.
- [ ] **Step 2 — Establish identity and scope:** record the agent's own identity, extract the principal/action/resource ARNs, and flag cross-account.
- [ ] **Step 3 — Collect evidence, policy reads first:** read the chain's policy documents by hand; use CloudTrail and simulation only as corroboration.
- [ ] **Step 4 — Walk the chain:** traverse the six hops in precedence order; do not stop at hop 1 just because it passed.
- [ ] **Step 5 — Apply service-specific knowledge:** rule out non-IAM denial causes for the service explicitly.
- [ ] **Step 6 — Assign verdicts:** give every hop exactly one token from the closed verdict set.
- [ ] **Step 7 — Propose a policy:** derive a scoped policy for human review; keep observed and commonly-required permissions labelled separately.
- [ ] **Step 8 — Deliver the report:** render per the report format, run the pre-render validation, then deliver.

## Output Discipline

The report is the deliverable. Conversation around it is not.

- **Do not narrate API calls.** No per-call summaries, no interim results, no raw response
  extracts. A full diagnosis makes many reads; announcing each one buries the finding.
- **Do not narrate plans or reasoning.** No "Let me check...", "I'll now look at...",
  "Given the chain, I should...". Execute the step and move on.
- **Do not echo raw API responses.** Process them silently. Policy documents in particular
  are long, and pasting them displaces the diagnosis.
- **Keep interstitial messages to one line.** Speak between steps only at real milestones:
  starting, asking the user something, delivering, or erroring.
- **Do not summarize after delivering.** The report already contains the summary;
  restating it invites a shortened paraphrase to be read instead of the report.
- **Never assess your own performance.** Do not append a paragraph saying the diagnosis
  worked, was correct, handled a hard case, or caught something subtle. The reader
  evaluates the report; the report does not evaluate itself. Self-congratulation also
  lends unearned confidence to findings whose limitations the report has just carefully
  enumerated.
- **Nothing follows the report** except, at most, a single line offering a next action —
  saving an artifact, or running another failure. No recap, no restatement of the root
  cause, no commentary on the diagnosis.

## Supported Services

| Service | Coverage |
|---|---|
| Amazon Bedrock | Full — including non-IAM denial causes |
| Amazon SageMaker | Full — including PassRole and execution-role chains |
| Other AI/ML services | Not supported in this version. State this plainly and stop. |

If the request concerns an unsupported service, say so and do not attempt a partial
diagnosis from the generic chain alone. The value of this skill is in the
service-specific knowledge; without it the output would be a guess.

## Architecture

- **This skill (orchestrator):** request classification, chain traversal order,
  verdict assignment, report rendering.
- **Chain model:** the six-hop authorization chain and its precedence rules —
  `references/access-chain-model.md`
- **Data collection:** the read-only API allowlist, error classification, and the
  structured object collection produces —
  `references/data-collection.md`
- **Finding logic:** verdict rules and body templates per failure class —
  `references/finding-logic.md`
- **Report format:** report structure and pre-render validation —
  `references/report-format.md`
- **Service specifics:** loaded only for the service in question —
  `references/svc-bedrock.md`,
  `references/svc-sagemaker.md`

## Step 1: Classify the request

**Classify before calling any tool.** Two things must be established first.

### 1a. Which service?

Determine the AI/ML service from the error text, API name, or resource ARN. If it is
not Bedrock or SageMaker, stop and report it as unsupported.

### 1b. Is there an observed failure?

| Evidence available | Route |
|---|---|
| User pasted an error message | **Observed** — parse it, then corroborate with CloudTrail |
| No error text, but a principal and action are named | **Observed** — locate the event in CloudTrail |
| Neither | **Stop.** Ask for the error message, or the principal ARN plus the API call that failed. |

This skill diagnoses failures. It does not audit permissions speculatively. If there
is no failure to explain, say so and stop rather than producing a posture review.

## Step 2: Establish identity and scope

1. Call `sts:GetCallerIdentity` to determine the account and the identity the agent
   itself is operating as. Record it — the report must state whose view this is.
2. From the error text, extract: the **principal ARN**, the **action**, and the
   **resource ARN** where present. Error strings of the form
   `User: <arn> is not authorized to perform: <action> on resource: <arn>` carry all
   three.
3. Determine whether the principal is in the current account. If the resource is in a
   different account, mark the request **cross-account** and follow the cross-account
   handling in `references/finding-logic.md`.

## Step 3: Collect evidence — policy reads first

**Policy documents are the primary evidence.** Every hop except the organization SCP
decision is decidable by reading the policies that govern it. CloudTrail and the policy
simulator are corroboration, and the diagnosis must stand without either — in this runtime
both are frequently unavailable, which is a characteristic of the environment rather than a
permission gap. See `references/data-collection.md`.

Collect in this order:

1. **The chain's policy documents.** The caller's identity policies, the target role's
   trust policy and permissions, relevant resource policies, and the attached SCPs.
   Evaluate each by hand: match the action, match the resource ARN including its account
   and region fields, and check every condition key against what the failing call
   supplied.
2. **CloudTrail, if the runtime permits it.** Adds independent confirmation of the event

From aws/tools-for-devops-agent, Apache-2.0.

More from aws/tools-for-devops-agent