cost-aware-llm-pipeline

Files (2)

• SKILL.md
• SKILL.md.lock

SKILL.md (first lines)

<!-- bodyRef: skills/sources/cost-aware-llm-pipeline.md -->

<!-- Generated by `harness skills bootstrap`. Edit to tune; re-run to regenerate. -->
<!-- Static skills point at live docs by design — do not hardwire version-pinned API here. -->

# Cost-aware LLM pipeline (LangGraph + Bedrock Converse)

This project generates reviewable marketing/sales/demo content from release diffs via four LangGraph graphs calling Amazon Bedrock (Converse API). LLM tokens are the dominant variable cost, and fan-out (parallel artifact generation) + a long, repeated evidence context multiply it. This skill is about spending the *least* tokens for the *required* quality: route to the cheapest model that passes the gate, cache the repeated context, never re-bill work you already did, and measure every call.

**Core principle:** cheapest-tier-first with an eval gate, escalate only on failure. Pay for the expensive model only on the prompts that actually need it.

## Pull current docs first

Ground every decision in these — APIs, model minimums, and pricing shift often. Do NOT trust memorized model IDs or token thresholds.

- Bedrock prompt caching (cachePoint, per-model token minimums, TTL, billing): https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html
- Bedrock Intelligent Prompt Routing (single-endpoint tier routing within a family): https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html
- Bedrock Converse API (usage block, inferenceConfig): https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html
- LangGraph node caching + durable execution (CachePolicy, compile-time cache): https://docs.langchain.com/oss/python/langgraph/graph-api
- CachePolicy reference (key_func, ttl): https://reference.langchain.com/python/langgraph/types/CachePolicy
- ChatBedrockConverse integration (cachePoint blocks, usage_metadata cache fields): https://docs.langchain.com/oss/python/integrations/chat/bedrock
- Bedrock pricing (cache read/write rates differ by model): https://aws.amazon.com/bedrock/pricing/

## Where the money goes in this stack

1. **Repeated evidence context.** Every artifact in `content_generation_graph` re-sends the same approved feature manifest + skill snapshots. Uncached, you pay full input price N times per release. → Bedrock prompt caching.
2. **Deterministic recompute.** Re-running a release job (to fix one artifact, or after a crash) re-bills extraction/clustering that didn't change. → LangGraph node caching + checkpoint resume.
3. **Over-powered model on easy prompts.** UI-string extraction and click-path JSON don't need a frontier model; blog/one-pager generation might. → tier routing.
4. **Fan-out throttling.** `generate_artifacts_parallel` hitting Bedrock at once → TPM throttle → retries → more spend + latency.
5. **Guardrails on throwaway drafts.** Guardrails bill per call; running them before the quality gate wastes spend.

## 1. Model-tier routing

Map each node to the cheapest tier that meets its bar; escalate on a measured failure, don't default high.

- **Per-node static tier** (simplest, recommended default): pick the model per LangGraph node. Deterministic extractors/validators → Haiku-class; clustering/scoring → Sonnet-class; long-form generation → Sonnet/Opus-class. With `ChatBedrockConverse`, set `model_id` per node instance — see the integration doc.
- **Escalation gate:** run cheap tier, score the output (deterministic checks + a cheap LLM judge), and only re-run on the expensive tier when the gate fails. This beats "always use the big model" because most prompts pass cheap.
- **Bedrock Intelligent Prompt Routing** when you want one endpoint to pick between two models *in the same family* by predicted quality vs. cost (default routers, or configured with a fallback model + `responseQualityDifference`). Note its limits below — it does NOT replace your own cross-family routing.
- Centralize tier choice in one config/module so model IDs aren't scattered; the spec already snapshots active skills — snapshot the model tier per run too for provenance and cost attribution.

## 2. Caching — three independent layers, use all three

**a) Bedrock prompt caching (token cost).** Insert a `cachePoint` after the *static* prefix (tools → system → stable evidence) and before variable content. Reused tokens are billed at a reduced read rate.
- Sections chain `tools → system → messages`; the token minimum is the cumulative total across them. Place stable content first.
- Per-model minimums and TTL differ — check the doc; below-minimum cache points silently no-op.
- With `ChatBedrockConverse`, add `{"cachePoint": {"type": "default"}}` content blocks; read `usage_metadata["input_token_details"]` (`cache_read`, `cache_creation`) to confirm hits.
- Highest leverage here: the parallel artifact fan-out sharing one evidence/system prefix within a single run.

**b) LangGraph node caching (recompute cost).** `builder.add_node(..., cache_policy=CachePolicy(ttl=..., key_func=...))` + `builder.compile(cache=InMemoryCache())`. Identical input within TTL returns instantly with `cached: True` and makes **zero** LLM calls.
- Default `key_func` pickle-hashes the *entire* node input — key it explicitly on the evidence/manifest hash so a volatile field (run_id, timestamp) doesn't bust it every run.

**c) Checkpointing / durable execution (don't re-bill on resume).** Checkpoints save at super-step (between-node) boundaries. On resume, completed nodes are skipped — but a node that crashed/interrupted mid-way re-runs from its start, re-billing every LLM call it already made. Keep expensive generation in its own node that completes *before* the interrupt; split multi-call nodes with `@task`.

## 3. Token-budget discipline

- **Measure first.** Read the Converse `usage` block / LangChain `usage_metadata` on every call. When caching is on, `inputTokens` excludes cached tokens — compute `total = inputTokens + cacheReadInputTokens + cacheWriteInputTokens` or your cost dashboard will lie.
- **Cap output:** set `inferenceConfig.maxTokens` per artifact type; don't let a one-pager run to 8k tokens.
- **Prune input:** send pruned/ranked evidence (pgvector top-k), not the raw bundle. The biggest input win is sending less, then caching what's left.
- **Prefer structured output** (tool/JSON schema) over free-form + re-parse — fewer retries, tighter tokens.
- **Budget per run:** track cumulative tokens against a per-release ceiling; surface it on the dashboard alongside provenance.

## 4. Cost-quality eval gates

- Gate cheap-tier output with deterministic checks (claim-to-evidence linkage, policy regex) *before* any LLM judge or Guardrails call — deterministic checks are free.
- Run **Bedrock Guardrails only on the final candidate path**, not on every draft/escalation attempt.
- Use a cheap LLM as the quality judge for escalation decisions; reserve the expensive model for regeneration, not judging.

## 5. Stack wiring notes

- **LangGraph graphs (this spec):** apply node caching to `extract_*`/`cluster_*` nodes (stable given evidence hash); apply Bedrock prompt caching across `generate_artifacts_parallel`; keep the three approval interrupts AFTER their expensive node so resume is free.
- **Fan-out safely:** prime the shared evidence cache with one call before parallel fan-out — cache hits don't count against rate limits, so this both warms the cache and dodges TPM throttling.
- **Next.js dashboard:** stream tokens (Converse stream) to the review UI for perceived latency; cost/token metrics belong in Aurora telemetry, not recomputed client-side.
- **Playwright/media:** the `media_generation_graph` LLM cost is small (click-path JSON) — route it to the cheapest tier and cache the validated click path.

## Checklist

- [ ] Pulled the current docs above; verified model IDs, cache minimums, and TTLs from them (not memory).
- [ ] Every node assigned an explicit, justified model tier in one central config.
- [ ] Escalation path is cheap-first + measured gate, not always-expensive.
- [ ] `cachePoint` placed after the static tools/system/evidence prefix; verified cache hits via `usage_metadata` cache fields.
- [ ] Static prefix actually exceeds the model's per-checkpoint token minimum (else it no-ops).
- [ ] Node `CachePolicy` uses a `key_func` over the evidence/manifest hash, not the full volatile state.
- [ ] Expensive generation completes before any interrupt; multi-call nodes split with `@task`.
- [ ] Token accounting includes cacheRead + cacheWrite, not just `inputTokens`.
- [ ] `maxTokens` capped per artifact type; input pruned to top-k before sending.
- [ ] Guardrails + LLM judge run only on the final/escalated path.
- [ ] Parallel fan-out primes the cache first and respects TPM limits.
- [ ] Per-run token budget tracked and surfaced; model tier snapshotted for provenance.

## Gotchas

- **Cache point below the model minimum silently does nothing.** Inference still succeeds, you just pay full price with no warning. A `cachePoint` after a short system prompt buys zero. Confirm the prefix clears the per-model minimum (1024 vs 4096 tokens depending on model — check the doc).
- **`inputTokens` excludes cached tokens in Converse.** Naive cost dashboards undercount badly. Always add `cacheReadInputTokens + cacheWriteInputTokens`.
- **Cache write can cost MORE than uncached input.** Caching content read fewer than ~2 times can be a net loss. Cache only genuinely repeated prefixes (the shared evidence across the parallel fan-out), not one-shot prompts.
- **Converse cache sections chain `tools → system → messages`; changing an earlier section invalidates all later caches.** A per-node dynamic tool list or a system prompt with an embedded timestamp/run_id kills every downstream cache hit. Keep tools and system byte-stable across the fan-out.
- **5-minute TTL is wall-clock and resets only on hit.** GitHub Actions release jobs run sporadically, so the cache is almost always cold *across* runs — prompt caching helps *within* a run (parallel artifacts reusing one evidence prefix), not between releases. 1-hour TTL exists only on some models (e.g. Opus 4.5 / Sonnet 4.5 / Haiku 4.5) — Opus 4.6 / Sonnet 4.6 are 5-minute only at time of writing; verify against the doc.
- **LangGraph default `key_func` pickle-hashes the entire node input.** Any volatile field in state (run_id, timestamp, ordering) busts the node cache every single run, so you think caching is on but pay full price. Always pass an explicit `key_func`.
- **LangGraph checkpoints at super-step boundaries, not mid-node.** A node that makes 3 Bedrock calls then interrupts re-runs all 3 (and re-bills) on resume. This directly bites the spec's three approval gates — put expensive generation in a node that *completes* before the interrupt, or split calls into `@task` units, or you regenerate (and re-charge) every artifact each time a reviewer resumes.
- **Intelligent Prompt Routing only routes within one model family and is English-only.** It cannot route Anthropic↔Nova, and can't adapt on your app-specific performance data. Cross-family cost routing must be your own LangGraph router node.
- **Prompt caching is on-demand only — not supported with Bedrock batch inference.** If you move the fan-out to batch for cost, you lose caching; weigh the tradeoff.
- **Guardrails bill and add latency per call.** Running them on cheap pre-filtered drafts or on every escalation attempt wastes money — gate them to the final approved-candidate path only.
- **`ChatBedrockConverse` won't insert cachePoints for you.** The `usage_metadata` cache fields stay zero unless you actually send `cachePoint` blocks. Seeing zeros usually means you forgot the blocks, not that caching failed.
- **Re-running the whole release job to fix one artifact re-bills everything.** Combine checkpoint resume + node cache keyed on the evidence hash so unchanged upstream extraction/clustering nodes don't recompute or re-call Bedrock.
- **Cross-region inference can increase cache writes** during high demand (it may route to a region with a cold cache), quietly raising cost. Watch `cacheWriteInputTokens` if you enable it.

<!-- sources fetched at generation: https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html, https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html, https://docs.langchain.com/oss/python/langgraph/graph-api, https://reference.langchain.com/python/langgraph/types/CachePolicy, https://docs.langchain.com/oss/python/integrations/chat/bedrock -->

From ichiorca/shipsignal, Apache-2.0.

More from ichiorca/shipsignal