agentic-engineering

Files (2)

• SKILL.md
• SKILL.md.lock

SKILL.md (first lines)

<!-- bodyRef: skills/sources/agentic-engineering.md -->

<!-- Generated by `harness skills bootstrap`. Edit to tune; re-run to regenerate. -->
<!-- Static skills point at live docs by design — do not hardwire version-pinned API here. -->

# Agentic Engineering (LangGraph / Bedrock / Next.js / Playwright)

Build agents the way this stack rewards: **eval-first**, **decomposed into small graphs**, with **cost-aware model routing**. Don't generate or act on model output you can't measure or roll back. This skill is concept-first — follow the doc links for current API details rather than trusting memorized signatures.

## Pull current docs first

LangGraph and the AWS/eval surfaces move fast. Before writing graph or eval code, fetch the relevant page(s):

- LangGraph graph & state API — https://docs.langchain.com/oss/python/langgraph/graph-api
- LangGraph human-in-the-loop (interrupts) — https://docs.langchain.com/oss/python/langgraph/interrupts
- LangGraph multi-agent / supervisor — https://reference.langchain.com/python/langgraph-supervisor
- LangGraph swarm / handoffs — https://github.com/langchain-ai/langgraph-swarm-py
- LangSmith evaluation — https://docs.langchain.com/langsmith/evaluation
- Agent trajectory evaluators — https://github.com/langchain-ai/agentevals
- Amazon Bedrock Converse API — https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html
- ChatBedrockConverse (langchain-aws) — https://reference.langchain.com/python/langchain-aws/chat_models/bedrock_converse/ChatBedrockConverse
- Playwright locators & auto-waiting — https://playwright.dev/docs/locators

## Eval-first execution

Write the check before the capability. An agent step is "done" only when a gate can prove it.

- **Three gate tiers**, cheapest first: (1) **deterministic** checks (schema validation, regex/string presence, "does the claim cite real evidence", does the click-path JSON validate); (2) **heuristic/code** evaluators; (3) **LLM-as-judge** only where 1–2 can't reach. Run cheap gates before spending tokens on expensive ones.
- Curate a **dataset** of representative inputs (real release diffs, edge cases) and run **offline evaluation** with `evaluate()` against it in CI; add **online evaluation** on live traffic later. See the LangSmith evaluation doc.
- For agents, evaluate the **trajectory** (tool calls + intermediate steps), not just the final string — use `agentevals` trajectory match (reference-based) or LLM-as-judge trajectory (reference-free).
- **Wire evals into CI as a regression gate**: LangSmith integrates with pytest / Vitest / GitHub workflows. Set score thresholds and fail the pipeline when they drop. A passing eval suite is the merge gate, the same way unit tests are.
- Snapshot the prompt/skill version used for each run so an eval regression maps to a specific change.

## Decomposition (small graphs, not one mega-agent)

- Prefer **several small, single-purpose graphs** wired by control flow over one monolithic agent — easier to eval, retry, and reason about. Compose them as **subgraphs** when they share state.
- Model state explicitly: a `TypedDict` (or Pydantic) **state schema** is the contract between every node. Updates are **partial** — a node returns only the keys it changes.
- Choose the multi-agent shape deliberately: **supervisor** (central router delegates via tool-based handoff) for controlled flows; **swarm** (agents hand off peer-to-peer, last-active agent remembered) for open-ended ones. Handoffs use `Command(goto=..., graph=Command.PARENT)`; the target's state schema must contain the keys you put in `Command.update`.
- For dynamic, data-dependent fan-out (one branch per feature/item), use the **Send API** (`Send(node, state)`) for map-reduce — not a fixed set of edges.

## Tool & approval wiring

- **Approval gates use `interrupt()`**, not custom polling. `interrupt(payload)` pauses the graph and surfaces a JSON-serializable payload to your UI; the reviewer's action resumes via `Command(resume=value)`, which becomes the return value of `interrupt()`. Interrupts **require a checkpointer + a `thread_id`** in config — resume the same `thread_id`.
- Keep tools small and typed; validate tool inputs/outputs deterministically so a bad call fails a gate instead of poisoning state.
- In this stack, run the **long graph job on the GitHub Actions runner**; let the **Next.js/Vercel API routes stay thin** (trigger workflow_dispatch, read state, resume a thread). Don't run a multi-minute graph inside a serverless request.

## Cost / latency control

- **Route by task difficulty**: cheap/fast model for extraction, classification, redaction, click-path generation; strong model only for synthesis and judging. With Bedrock this is just a different `modelId` per node via `ChatBedrockConverse`.
- Use **`ConverseStream`** for user-facing latency; plain `Converse` for batch/back-end steps.
- Enable **prompt caching** for stable system prompts / large shared context; track token + cost per node so routing decisions are data-driven.
- Prefer deterministic extractors over LLM calls wherever a parser will do — cheapest token is the one you don't spend.

## Playwright (deterministic capture)

- Drive flows with **user-facing, auto-waiting locators**: `getByRole`, `getByLabel`, `getByText`, `getByTestId`. These survive DOM churn and remove manual waits. Avoid brittle CSS/XPath chains tied to DOM structure.
- Generate a **validated click-path** (schema-check the JSON) before running capture; record video/screenshots as evidence artifacts.

## Checklist

- [ ] Fetched the current doc pages above for the APIs you're touching.
- [ ] State schema defined as a TypedDict with explicit reducers for any list/accumulator channel.
- [ ] Work split into small graphs/subgraphs; multi-agent shape (supervisor vs swarm) chosen on purpose.
- [ ] Parallel/fan-out branches use `Send` and have reducers — no silent overwrite of shared keys.
- [ ] Approval points use `interrupt()` + `Command(resume=...)` with a checkpointer and a stable `thread_id`.
- [ ] Side-effects sit *after* interrupts (or are idempotent) — see Gotchas.
- [ ] Eval dataset + deterministic/heuristic/LLM-judge gates exist; CI fails on threshold regression.
- [ ] Model routing: cheap model for mechanical steps, strong model for synthesis/judging; streaming where latency matters.
- [ ] Long jobs run on the Actions runner; Next.js routes only trigger/resume/read.

## Gotchas

- **Resuming re-runs the whole node from its top — not from the `interrupt()` line.** Any code before `interrupt()` executes again on every resume. Put DB writes, S3 uploads, and other non-idempotent side-effects *after* the interrupt or in a separate downstream node, or make them idempotent. This is the #1 LangGraph HITL bug.
- **Resume values are matched to interrupts by index, deterministically.** Multiple `interrupt()` calls in one node, or interrupts inside data-dependent loops/conditionals, can mis-map resume values. Keep interrupt order stable.
- **Never wrap `interrupt()` in a bare `try/except`.** It raises a special control-flow exception; a broad handler swallows it and the pause silently breaks.
- **No checkpointer ⇒ no interrupts and no durable state.** Use an in-memory checkpointer only for tests; production needs a durable one (e.g. Postgres). Forgetting `thread_id` means resume can't find the run.
- **Parallel branches without a reducer overwrite each other.** When fan-out nodes write the same state key, the default reducer keeps only the last write — results vanish silently. Annotate accumulator channels (e.g. `Annotated[list, add]` / `add_messages`).
- **Bedrock inference-profile vs model-id mismatch.** Cross-region inference is done via an inference-profile ARN, but AWS gives you no API to read the underlying model back from the profile — you must keep the `model` param in sync with the profile yourself, or metadata/routing silently diverges. Also: a model returning `AccessDenied` usually means model access isn't enabled in that region, not a code bug.
- **`Converse` tool-use and message shape differ per model family.** The Converse API normalizes a lot, but tool-call formats and system-prompt handling still vary by underlying model — re-check after switching `modelId` in a routing change.
- **LLM-as-judge evals are nondeterministic.** Pin the judge model + prompt, run multiple repetitions, and treat a small score delta as noise — otherwise CI flakes. Cache eval runs where the doc allows to control cost.
- **Next.js serverless timeouts kill long graph runs.** A graph that analyzes diffs, calls Bedrock, and runs Playwright will exceed Vercel function limits — run it on the Actions runner and have the route only dispatch/resume/poll.
- **Brittle Playwright selectors break demo capture on any UI tweak.** Prefer role/label/text/testId locators; a CSS `nth-child` chain will silently capture the wrong element after a re-render.

<!-- sources fetched at generation: https://docs.langchain.com/oss/python/langgraph/interrupts, https://docs.langchain.com/oss/python/langgraph/graph-api, https://docs.langchain.com/langsmith/evaluation, https://playwright.dev/docs/locators, https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html, https://reference.langchain.com/python/langgraph-supervisor, https://github.com/langchain-ai/langgraph-swarm-py, https://github.com/langchain-ai/agentevals, https://reference.langchain.com/python/langchain-aws/chat_models/bedrock_converse/ChatBedrockConverse -->

From ichiorca/shipsignal, Apache-2.0.

More from ichiorca/shipsignal