investigation-cost-guardrail

Files (9)

• CHANGELOG.md
• README.md
• SKILL.md
• benchmark.json · evals/
• eval_queries.json · evals/
• evals.json · evals/
• report.json · evals/
• trigger_report.json · evals/
• pricing-reference.md · references/

SKILL.md (first lines)

# Investigation Cost Guardrail Skill

## Overview

This skill provides cost guardrails for ANY AWS service and ALL native agent tools — not just a hardcoded list. It uses **heuristic classification** to determine whether an API operation is free or paid, estimates cost before execution, and enforces per-investigation budgets.

## Design Principle

Rather than listing every free/paid operation across 200+ AWS services, this skill uses three layers:

1. **Heuristic rules** — classify any operation based on naming patterns and behavior
2. **Known-paid registry** — explicit overrides for high-cost operations with pricing formulas
3. **Response validation** — detect metered usage from API response fields after execution

## Activation

This skill MUST be ALWAYS ACTIVE during investigations. It does NOT require user invocation.

## Fetch Live Rate Before Estimating

The **first time** an operation is classified **PAID** by Layer 2, fetch the live rate **before** estimating cost.

For **AWS operations**: read `references/pricing-reference.md` for the exact Pricing API call patterns, filter fields, filter values, and failure handling. The reference file specifies — for each service and operation — whether to use `Field:"operation"` or `Field:"usagetype"`, and the exact value to use. Do not derive these from the operation name.

For **non-AWS tools** (Splunk, Datadog, Grafana, etc.): use the cost model from Layer 0 directly — no live lookup available.

---

## Layer 0: Native Agent Tool Classification

Before Layer 1 heuristics, classify the agent's own tools. These are NOT `use_aws` calls but have distinct billing implications:

### Tool Cost Matrix

| Tool | Classification | Cost Model | Guardrail |
|---|---|---|---|
| `get_prometheus_metrics` | **PAID** | Billed per sample scanned — **rate from live `CW:PromQL:SamplesScanned` usagetype lookup** (`AmazonCloudWatch`, workload-region prefix) | Track samples scanned per call; HALT if rate lookup returns 0 products |
| `use_aws` | **VARIABLE** | Depends on operation — apply Layers 1–3 | Full heuristic pipeline |
| `use_azure` | **FREE** | Azure Reader role, no per-call billing | Track count only |
| `grafana_query_prometheus` | **CAUTION** | Depends on Grafana data source billing model | Track count, warn at 50+ |
| `use_datadog` | **CAUTION** | Datadog API rate limits (no per-call $ cost, but may throttle) | Track count, warn at 100+ |
| `use_splunk` | **PAID** | Splunk search license (per GB ingested/searched) | Treat like CW Logs StartQuery |
| `use_pagerduty` | **FREE** | PagerDuty API (rate limited, not per-call billed) | Track count only |
| `shell` | **CAUTION** | May invoke `aws`, `az`, `kubectl` — untracked by Layers 1–3 | Log commands, warn if aws/az detected |
| `subagent` | **PAID** | Counts toward agent-seconds billing ($0.0083/sec) | Track spawns, enforce total time |
| `fs_read`, `fs_write`, `fs_tree` | **FREE** | Local file I/O | No guardrail needed |
| `datetime` | **FREE** | Internal state ops | No guardrail needed |
| `write_scratchpad`, `read_scratchpad` | **FREE** | Internal state (may not be available in all environments) | No guardrail needed |
| `read_memories` | **FREE** | Internal memory recall | No guardrail needed |

### PromQL-Specific Controls

`get_prometheus_metrics` deserves special handling because:

- Maximum 500 series per query — a broad query hitting the cap costs `500 × rate` per period.
- Range queries with small `step` multiply cost: `7d / 60s step = 10,080 datapoints × 500 series = 5M samples`

**Before each PromQL call:**

```text
rate = live usagetype lookup (AmazonCloudWatch, usagetype=<PREFIX>-CW:PromQL:SamplesScanned)
       # bare "CW:PromQL:SamplesScanned" for us-east-1; use workload-region prefix for all others
if rate lookup returns 0 products: 🚫 HALT — do not hardcode or improvise a rate

estimated_samples = min(500, estimated_series) × (time_range_seconds / step_seconds)
estimated_cost = estimated_samples × rate

if estimated_cost > $0.50:
    ⚠️ WARN — suggest narrower time range, larger step, or label filters
if estimated_cost > $2.00:
    🚫 HALT — require approval or suggest aggregation (sum, topk, avg)
```

**Cost reduction for PromQL:**

- Use `sum by (label)` to reduce series count
- Use `topk(N, ...)` to cap returned series
- Increase `step` (300s instead of 60s = 5× cheaper)
- Narrow time range (1h instead of 7d = 168× cheaper)

---

## Layer 1: Heuristic Classification

Before making ANY `use_aws` call, classify the operation using these rules IN ORDER:

### Rule 1: FREE by default — Metadata operations

An operation is FREE if it matches ALL of these:

- Verb is: `Describe`, `List`, `Get`, `Lookup`, `Check`, `Validate`, `Tag`, `Untag`
- It returns metadata/configuration (not data content or query results)
- It does NOT scan, process, or transform customer data

⚠️ Exception: Some services charge per-request even for Get/List operations. Layer 2 overrides this heuristic for S3 and Lambda Invoke — when Layer 2 has an entry, it takes precedence over Rule 1.

> ⚠️ **Tool policy can override cost classification.** Some operations classified as FREE here (e.g., `cloudtrail:LookupEvents`) may be blocked by tool policy in certain environments. If an operation is denied, it costs $0.00 (never executed) — proceed with alternatives.

### Rule 2: PAID — Data-scanning operations

An operation is PAID if it matches ANY of these patterns:

| Pattern | Why It Costs Money | Examples |
|---|---|---|
| Verb contains `Query` | Scans indexed data | `StartQuery`, `StartQueryExecution` |
| Verb contains `Scan` | Full table/index scan | `Scan` (DynamoDB), `StartScan` |
| Verb contains `Execute` + processes data | Runs a computation | `StartQueryExecution` (Athena), `ExecuteStatement` |
| Verb contains `Invoke` + runs workload | Triggers compute | `InvokeEndpoint` (SageMaker), `Invoke` (Lambda) |
| Operation reads **content** (not metadata) | Data transfer | `GetObject` (S3, large), `GetLogEvents` (bulk), `BatchGetTraces` |
| Operation starts a **streaming session** | Per-time billing | `StartLiveTail` |
| Operation name contains `Insights` | Analytics processing | `GetContributorInsights`, `GetInsightRuleReport` |

### Rule 3: CAUTION — High-volume free operations

An operation is FREE but CAUTION if:

- It's a paginated List/Describe that could return thousands of results
- It has no built-in limit and the scope is broad (e.g., all resources in a region)

**Examples:** `ListObjectsV2` (large bucket), `ListMetrics` (unfiltered), `DescribeTasks` (large cluster)

From aws/tools-for-devops-agent, Apache-2.0.

More from aws/tools-for-devops-agent