devops-agent-cost-insights

Files (131)

• CHANGELOG.md
• README.md
• SKILL.md
• benchmark.json · evals/best-practices/v1/
• best-practices-tests-results.json · evals/best-practices/v1/iteration-1/
• best-practices-tests-results.json · evals/best-practices/v1/iteration-2/
• best-practices-tests-results.json · evals/best-practices/v1/iteration-3/
• evals.json · evals/
• benchmark.json · evals/functional/v2/
• evals.json · evals/functional/v2/
• functional-tests-results.json · evals/functional/v2/iteration-1/athena-price-only-exact-role-joined-execution/with_skill/
• journal_records.json · evals/functional/v2/iteration-1/athena-price-only-exact-role-joined-execution/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-1/athena-price-only-exact-role-joined-execution/without_skill/
• journal_records.json · evals/functional/v2/iteration-1/athena-price-only-exact-role-joined-execution/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-1/cost-report-never-starts-a-query/with_skill/
• journal_records.json · evals/functional/v2/iteration-1/cost-report-never-starts-a-query/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-1/cost-report-never-starts-a-query/without_skill/
• journal_records.json · evals/functional/v2/iteration-1/cost-report-never-starts-a-query/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-1/direct-usage-four-metrics-from-supplied-data/with_skill/
• journal_records.json · evals/functional/v2/iteration-1/direct-usage-four-metrics-from-supplied-data/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-1/direct-usage-four-metrics-from-supplied-data/without_skill/
• journal_records.json · evals/functional/v2/iteration-1/direct-usage-four-metrics-from-supplied-data/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-1/logs-pagination-before-exact-attribution/with_skill/
• journal_records.json · evals/functional/v2/iteration-1/logs-pagination-before-exact-attribution/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-1/logs-pagination-before-exact-attribution/without_skill/
• journal_records.json · evals/functional/v2/iteration-1/logs-pagination-before-exact-attribution/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-1/partial-window-space-credit-is-not-allocated/with_skill/
• journal_records.json · evals/functional/v2/iteration-1/partial-window-space-credit-is-not-allocated/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-1/partial-window-space-credit-is-not-allocated/without_skill/
• journal_records.json · evals/functional/v2/iteration-1/partial-window-space-credit-is-not-allocated/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-1/per-investigation-direct-cost-is-not-inferred/with_skill/
• journal_records.json · evals/functional/v2/iteration-1/per-investigation-direct-cost-is-not-inferred/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-1/per-investigation-direct-cost-is-not-inferred/without_skill/
• journal_records.json · evals/functional/v2/iteration-1/per-investigation-direct-cost-is-not-inferred/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/athena-price-only-exact-role-joined-execution/with_skill/
• journal_records.json · evals/functional/v2/iteration-2/athena-price-only-exact-role-joined-execution/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/athena-price-only-exact-role-joined-execution/without_skill/
• journal_records.json · evals/functional/v2/iteration-2/athena-price-only-exact-role-joined-execution/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/cost-report-never-starts-a-query/with_skill/
• journal_records.json · evals/functional/v2/iteration-2/cost-report-never-starts-a-query/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/cost-report-never-starts-a-query/without_skill/
• journal_records.json · evals/functional/v2/iteration-2/cost-report-never-starts-a-query/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/direct-usage-four-metrics-from-supplied-data/with_skill/
• journal_records.json · evals/functional/v2/iteration-2/direct-usage-four-metrics-from-supplied-data/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/direct-usage-four-metrics-from-supplied-data/without_skill/
• journal_records.json · evals/functional/v2/iteration-2/direct-usage-four-metrics-from-supplied-data/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/logs-pagination-before-exact-attribution/with_skill/
• journal_records.json · evals/functional/v2/iteration-2/logs-pagination-before-exact-attribution/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/logs-pagination-before-exact-attribution/without_skill/
• journal_records.json · evals/functional/v2/iteration-2/logs-pagination-before-exact-attribution/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/partial-window-space-credit-is-not-allocated/with_skill/
• journal_records.json · evals/functional/v2/iteration-2/partial-window-space-credit-is-not-allocated/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/partial-window-space-credit-is-not-allocated/without_skill/
• journal_records.json · evals/functional/v2/iteration-2/partial-window-space-credit-is-not-allocated/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/per-investigation-direct-cost-is-not-inferred/with_skill/
• journal_records.json · evals/functional/v2/iteration-2/per-investigation-direct-cost-is-not-inferred/with_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-2/per-investigation-direct-cost-is-not-inferred/without_skill/
• journal_records.json · evals/functional/v2/iteration-2/per-investigation-direct-cost-is-not-inferred/without_skill/outputs/
• functional-tests-results.json · evals/functional/v2/iteration-3/athena-price-only-exact-role-joined-execution/with_skill/
• journal_records.json · evals/functional/v2/iteration-3/athena-price-only-exact-role-joined-execution/with_skill/outputs/
• …and 71 more

SKILL.md (first lines)

## Overview

This skill provides the **methodology** for gathering and calculating AWS DevOps Agent cost data. It intentionally does NOT prescribe a fixed output format. The consuming agent (chat, custom agent, etc.) decides its own presentation (plain text, multi-space tables, single-space visual dashboard with charts, etc.) based on its own prompt/output spec. Follow the steps below to gather and compute correct numbers; format them however the calling context requires.

### Execution visibility

The consuming agent's output contract controls visibility. When it requires silent, final-only execution:

- Perform skill/reference reads, planning, tool calls, pagination, distillation, calculations, retries, and Step 7 validation without assistant-authored progress messages.
- Keep checklist state, intermediate values, query IDs, raw records, arithmetic, and rejected records internal.
- Do not announce file reads or tool calls, react to results, or narrate what happens next.
- Return only the consuming agent's single final response after its artifact operation. A compact validation result table is allowed only when its final template explicitly requires one.

> **This skill must add exactly $0.00 to the bill.** Never `StartQuery`, `StartQueryExecution`, or `GetMetricData` — and never any other operation the **zero-cost rule** names below. Each is a paid API, so issuing one bills the account this report is about. If a figure is only reachable through a paid call, it is unavailable; say so. This is the **zero-cost rule**, restated at Step 1 and attested in Step 7.
>
> **"Read-only" is not the safety test.** `GetMetricData` reads only your own metrics and is billed per metric anyway. Before using any call not already named in the steps below, classify it per *Operation classification* in Step 3; an operation you cannot confirm as free is unclassified, not free.

Four things to cover every time:
1. **Discovery**: enumerate Agent Space(s) and DevOps Agent role(s) in scope (may be one space or all spaces in the account, depending on what's requested)
2. **Direct usage**: agent-seconds billed (investigations, evaluations, chat), per space
3. **Downstream**: AWS API costs from agent-initiated calls (Logs Insights, Athena, X-Ray, CloudWatch), attributed via CloudTrail, traceable by query ID / query execution ID
4. **Grouping**: if multiple spaces/roles are in scope, group cost by Agent Space ID and by DevOps Agent Role ARN; if scoped to a single space/role, a single summary suffices

---

## Workflow Checklist

Work through this checklist in order and do not skip a step. The steps are dependent, not independent: Step 1 fixes the `<START>`/`<END>` window that Steps 3 and 7 must reuse, and Step 7 validates the figures produced by Steps 1 through 6. Track checklist completion internally. Do not print step-by-step progress or skipped-step narration; surface only trust-impacting omissions in the consuming agent's single final response.

- [ ] Step 0: Discover the Agent Space(s) and DevOps Agent role(s) in scope. Skip only if scope is already fixed to one space/role
- [ ] Step 1: Resolve the reporting window from the request, then compute direct usage (agent-seconds) via CloudWatch `GetMetricStatistics`
- [ ] Step 2: Apply Support plan credits to get net direct cost
- [ ] Step 3: Attribute downstream cost using journal for call counts (primary), history APIs for exact bytes, and CloudTrail for cross-check, reusing the exact Step 1 window
- [ ] Step 4: Group by Agent Space ID and by Role ARN. Skip if scope is a single space/role
- [ ] Step 5: Extract tool usage from journal records and cross-reference against CloudTrail
- [ ] Step 6: Produce a per-investigation breakdown, only when requested
- [ ] Step 7: Validate every computed figure before reporting it

### Scope precedence

The consuming agent defines the scope. When it supplies an explicit `ROLE_ARN` and declares exact-role scope, that scope overrides this skill's multi-role discovery and role-coverage guidance:

- Treat `ROLE_ARN` as the complete allowlist of one role. Skip role discovery and never add historical, rotated, inferred, or similarly named roles.
- Require exact account and role identity. For CloudTrail, `userIdentity.sessionContext.sessionIssuer.arn` must equal `ROLE_ARN`. For an STS assumed-role identity, normalize only an ARN with the same account and exact configured role name to `ROLE_ARN`.
- Discard every other role immediately. Do not count, analyze, name, summarize, or mention excluded roles.
- Never widen exact-role scope because direct Agent Space usage exists on days with no matching downstream activity.
- Label direct usage as Agent Space-scoped because the `AgentSpaceUUID` metric cannot isolate one role. Label downstream and tool usage as exact-role-scoped. Do not present the combined result as the complete Agent Space total.

The multi-role discovery guidance below applies only when the consuming agent requests Agent Space-wide or account-wide scope.

### Definitions

These named rules are defined once here and referenced by name throughout. "Per the quantity source rule", "passes the attribution test", "per the Logs source rule", and "match the active role scope" mean exactly what is written below; the step does not restate them.

- **Active role scope:** the identity a record must match to count (see *Scope precedence*): the single configured `ROLE_ARN` in exact-role mode, or one of the Step 0 discovered role ARNs in multi-role mode.
- **Attribution test:** a record counts only when it matches BOTH the reporting window AND the active role scope; one alone is not enough. In exact-role mode, discard non-matching roles without analysis or mention. Failing this test is a leading source of run-to-run drift.
- **Logs source rule:** for CloudWatch Logs Insights, `describe_queries` is the sole source of query count, role attribution, scan bytes, and cost. Never substitute a CloudTrail `StartQuery` lookup; journal data is a cross-check only. If `describe_queries` is unavailable, report Logs Insights unavailable rather than substituting another source.
- **Identity-not-Username rule:** attribute on the parsed `userIdentity` / `sessionIssuer.arn`, never on a CloudTrail `Username` filter. DevOps Agent roles in one account share a single CloudTrail session name, so a `Username` count aggregates across roles and overstates the target. See *Cross-check: CloudTrail* for the mechanics.
- **CloudTrail-lower-bound rule:** CloudTrail Event History does not reliably return every matching event, so a CloudTrail-derived count is a lower bound, never an exact total. This is why the journal is primary.
- **Quantity source rule:** the Pricing API supplies a rate, never a quantity. Which quantity an operation needs follows from the **unit class** of its price dimension, not its service. A **count-class** unit (`Requests`, `Metrics`, `traces`) prices from the attributed call count, which Step 3 produces for every operation, so any per-call price dimension can be priced, including one from a service named nowhere in this skill; where a service tiers its requests, name the tier or do not price. A **data-class** unit (`GB`, `TB`) needs a byte figure, so it prices only from the sources Step 3 names, never from a call count. When the quantity is missing, report the call count with no dollars: these are all **paid** operations, so a row without dollars is a measurement gap, never a free call, and makes the Priced Downstream Total a **lower bound**. A confirmed `0` calls is exactly `$0.00`, not a gap. Each is countable when the audited agent issued it, forbidden when this run would (see *Zero-cost rule*).
- **Zero-cost rule:** this run must cost $0.00: no `StartQuery`, no `StartQueryExecution`, no `GetMetricData`, no log scanning. Use only the confirmed-free calls named in these steps. If a step appears to need a billable call, it finishes without that figure and reports the gap. Defined in the Overview, restated at the point of temptation in Step 1, and attested in Step 7. The rule binds by effect, not by name, across three evasion paths:
  - **Wrappers and aliases count.** The prohibition is on the paid operation, whatever name reaches it. Do not issue the canonical operations (`StartQuery`, `StartQueryExecution`, `GetMetricData`, `BatchGetTraces`) or any wrapper or tool form of them, including `start_query`, `query_cloudwatch_logs`, `get_metric_data`. Do not call `GetQueryResults` either: it is free, but it is the second half of a `StartQuery` pair and `describe_queries` already returns `bytesScanned`. An operation not named here is classified per *Operation classification* in Step 3 before it is called, and is unclassified rather than free if it cannot be confirmed free.
  - **`shell` is not an escape hatch.** A paid operation invoked through `shell` (for example `aws logs start-query`, `aws cloudwatch get-metric-data`, or an equivalent CLI or script) bills the account exactly as the tool form would. Do not use `shell`, `use_aws`, or any command runner to issue a scan or paid read this rule forbids by name.
  - **Delegation inherits the rule.** A delegated agent or subagent acts on the audited account's behalf, so a scan it issues is billed the same. Carry this rule into any delegated request explicitly, and never delegate an action you could not issue directly. Journal access does not normally require delegation: `get_investigation_journal_records` and `get_other_agentspace_journal_records` are called directly. When those tools are blocked this run, the Context Gatherer subagent is the fallback (see Step 3), and this rule travels with the delegated request.

---

## Step 0: Discover Spaces and Roles (skip if scope is already fixed to one space/role)

What this step resolves depends on the scope the consuming agent asked for (see *Scope precedence* above): the single space this run is scoped to, or every space in the account. Nothing in it is inferred.

**Always — resolve the space this run is running in, and its role.** This is the whole purpose of the step when the consuming agent supplied no identifiers.

- `get_agent_space` for the current space returns its `name` and `agentSpaceId`.
- `list_associations` for that space returns its AWS configuration. Take the role from `configuration.aws.assumableRoleArn`, with `accountId` and `accountType` alongside it. Pair on this field and nothing else.

For a single-space request that is the whole step. Everything below applies **only** when the consuming agent requested Agent Space-wide or account-wide scope.

**Account-wide or Agent Space-wide scope — enumerate the remaining spaces.**

- `list_agent_spaces` returns every Agent Space in the account, each with `name`, `agentSpaceId`, and `createdAt`. This is the authoritative inventory of *live* spaces.
- Call `list_associations` for each one and read `configuration.aws.assumableRoleArn` the same way.

> **Never pair a role to a space by creation timestamp.** Role and space creation times are not correlated: a space's role may be created long after the space, while an unrelated role is created moments before it. Nearest-timestamp pairing therefore produces confident but wrong pairings, attributing one space's downstream cost to another space's role. `configuration.aws.assumableRoleArn` is the only pairing source.

### Reconcile live spaces against the spaces in the metrics (account-wide scope only)

`list_agent_spaces` returns spaces that exist now. CloudWatch retains `AgentSpaceUUID` dimension values for spaces that have since been deleted, so the metric namespace routinely carries many more space IDs than the account has spaces. Collect the `AgentSpaceUUID` values Step 1 will read and split them:

- **Live** — present in `list_agent_spaces`. Name and role are both known.
- **Retired** — present in the metrics only. The billed usage is real and must be counted, but the space cannot be named and has no association from which to resolve a role.

Report retired spaces as one `retired spaces` line with their count. Never fold them into a named space, and never drop them. An account-wide total that omits them is a **lower bound**; one that includes them is complete for direct usage only, because their downstream activity has no resolvable role. State which of the two you produced — Step 2's credit gate depends on it.

### Anomalies to report separately (account-wide or Agent Space-wide scope only)

- **Cross-account roles.** A trust policy names the account permitted to use the role in `aws:SourceAccount`. When that value is not the audited account, the role serves another account's space: exclude it from totals and report it as a cross-account anomaly. Read `aws:SourceAccount` for this and nothing else — see the warning below before reading `aws:SourceArn`.
- **Orphan roles.** An agent role that no live space's `assumableRoleArn` points at is residue from a deleted space. List these separately, and never pair one to a live space to make the counts reconcile. Apply this only to the agent role type below; the web app role is never an `assumableRoleArn`, so testing it this way reports every one of them as an orphan.

> **A trust policy does not pair a role to a space.** On the agent role, `aws:SourceArn` is normally the wildcard `agentspace/*` (no space ID); where a specific `agentspace/<id>` appears it is usually a web app role, which must never be paired. `aws:SourceArn` therefore fails on the role that matters and succeeds on the wrong one.

**Role types are not interchangeable, and names are not a discovery mechanism.** One space commonly has several roles with unrelated suffixes:

- `DevOpsAgentRole-AgentSpace-<suffix>` — assumed by the agent to reach your resources. Cost attribution uses this one only, and only when `configuration.aws.assumableRoleArn` names it.
- `DevOpsAgentRole-WebappAdmin-<suffix>` — enables the off-console web app. Never a cost-attribution role.

Paths vary (`role/...` and `role/service-role/...`), other patterns exist, and the suffixes of one space's roles do not match each other. Use names only to find orphans.

---

## Step 1: Direct Usage (CloudWatch GetMetricStatistics)

**The reporting window is an input, not a fixed period.** Take `<START>`/`<END>` from the request — an explicit range, a named period ("last 7 days", "August"), or the consuming agent's configured default. This skill prescribes no window of its own; a custom agent that wants a standing trailing window pins it in its own configuration and passes it in. Resolve it to absolute UTC timestamps once, here, then reuse those exact values everywhere downstream: Steps 3 and 7 depend on one window for the whole report, and a report that mixes windows describes no real period. Where scope affects credit eligibility (Step 2) or exactness labels (Step 7), those rules read the window resolved here.

Read metrics with `GetMetricStatistics`, never `GetMetricData` (**zero-cost rule**).

### Discover the activity metrics, do not assume them

**Enumerate the namespace first; never work from a fixed list of activity types.** Billable activities are whatever `AWS/AIDevOps` currently publishes; a hardcoded list silently drops any activity added later and understates the total with no error.

From aws/tools-for-devops-agent, Apache-2.0.

More from aws/tools-for-devops-agent