eval-harness

Files (2)

• SKILL.md
• SKILL.md.lock

SKILL.md (first lines)

<!-- bodyRef: skills/sources/eval-harness.md -->

<!-- Generated by `harness skills bootstrap`. Edit to tune; re-run to regenerate. -->
<!-- Static skills point at live docs by design — do not hardwire version-pinned API here. -->

# Eval Harness (LangGraph + LangSmith + Playwright)

Build **layered, gated** evals for an agentic system. The goal is not a single score — it is a pyramid that catches regressions cheaply at the bottom and expensively (rarely) at the top, plus **gates** that fail CI when quality, cost, or latency drift.

This stack: **LangGraph** (Python + TS) orchestration, **Bedrock Converse** as the model gateway, **Next.js/React** dashboard, **Playwright** for browser capture & e2e. Evals run in **GitHub Actions**.

## Pull current docs first
This space moves fast — read before you write. Don't trust pinned API from memory.
- Agent evals overview & `agentevals`: https://docs.langchain.com/oss/python/langchain/test/evals
- `agentevals` (trajectory + **graph trajectory** evaluators): https://github.com/langchain-ai/agentevals
- Run evals with pytest (CI gate): https://docs.langchain.com/langsmith/pytest
- Vitest/JS evals: https://blog.langchain.com/pytest-and-vitest-for-langsmith-evals/
- LangGraph interrupts / human-in-the-loop: https://docs.langchain.com/oss/python/langgraph/interrupts
- Multi-agent (supervisor/swarm/handoff): https://reference.langchain.com/python/langgraph-supervisor , https://reference.langchain.com/python/langgraph-swarm
- Cost & token tracking: https://docs.langchain.com/langsmith/cost-tracking
- Playwright testing: https://playwright.dev/docs/test-assertions

## Mental model: the eval pyramid
1. **Node/unit tests** — call a single graph node as a pure function over state. Fast, deterministic, no LLM. Assert state reducers, routing conditions (`tools_condition`), and tool wiring (`ToolNode`). These are your first gate.
2. **Trajectory evals** — run the graph end-to-end on a dataset and score the *sequence of nodes/tool calls*, not just the final answer. Use `agentevals`:
   - Deterministic: `create_trajectory_match_evaluator(trajectory_match_mode=...)` — modes `strict | unordered | subset | superset`. For LangGraph node-sequence form, use the **graph** variants (`create_graph_trajectory_match_evaluator`) and extract with `extract_langgraph_trajectory_from_thread` (handles `__interrupt__` pauses).
   - Qualitative: `create_trajectory_llm_as_judge` / `create_graph_trajectory_llm_as_judge`.
   - Tool-call correctness: stack `tool_args_match` + `correctness` — checks both *which/what args* and *was the answer useful*.
3. **End-to-end / UI evals** — Playwright drives the Next.js dashboard through real approval flows (the human gates) and asserts rendered artifacts/claims.

## Designing the graph FOR evaluability
Evals are only as good as the seams you give them.
- **Explicit state, typed reducers.** Use `StateGraph` with `Annotated` reducers. A node that mutates state opaquely can't be unit-tested; make each node a deterministic `state -> partial-state` function and push LLM calls behind injectable clients.
- **Checkpointer + `thread_id` are test infrastructure.** Trajectory extraction and interrupt/resume both require a checkpointer and a stable `thread_id`. Use an in-memory checkpointer in tests; reuse a `thread_id` to resume, new value = fresh state.
- **Handoffs as assertable edges.** Prefer the **supervisor** pattern when you need evals to be legible — every routing decision is one visible node in the trajectory. Swarm/`Command(goto=..., graph=Command.PARENT)` is faster but spreads routing across handoff tools; if you use it, evaluate handoff correctness explicitly.
- **Tool wiring.** Keep tools deterministic and side-effect-isolated so eval runs don't hit live services; assert `ToolNode` selection separately from tool execution.

## Testing human-approval gates (interrupt/resume)
The spec has 3 interrupt gates. Eval them by: invoke graph → assert `result['__interrupt__']` payload shape (gate id, `thread_id`, dashboard fields) → resume with `Command(resume=...)` on the same `thread_id` → assert downstream state. Cover approve / edit / reject branches. Trajectory extractors mark these `__interrupt__` — don't let them break strict-match comparisons.

## Eval gates in CI (the actual gate)
- **Python:** `langsmith>=0.3.4`, `pip install "langsmith[pytest]"`. Decorate cases `@pytest.mark.langsmith`; log with `t.log_inputs/log_outputs/log_reference_outputs/log_feedback(key, score)`. Use `expect(...).to_contain(...)` / `expect.embedding_distance(...).to_be_less_than(t)` to both record feedback **and** fail the test. A failing assertion = failing build = blocked merge. Wrap LLM-as-judge calls in `trace_feedback()` so feedback lands in scope.
- **TS:** mirror with the Vitest integration for the Next.js/React side.
- **Determinism & cost in CI:** cache LLM calls to cassettes (`LANGSMITH_TEST_CACHE=tests/cassettes`), commit them so CI replays instead of paying per run. Set `LANGSMITH_TEST_SUITE` / `LANGSMITH_EXPERIMENT` to compare against a baseline. Gate on aggregate experiment scores crossing thresholds, not just per-test pass.

## Cost & latency control
- Treat cost/latency as **first-class eval metrics with budgets**, asserted like correctness (`expect(tokens).to_be_less_than(...)`).
- LangSmith auto-records tokens/cost for OpenAI/Anthropic. **Bedrock Converse (this stack) is not auto-priced** — see Gotchas.
- Tag runs with metadata (feature, release_run_id, environment) for per-feature attribution; track P50/P99 latency on dashboards.
- Pick cheap models for deterministic checks; reserve LLM-as-judge for trajectory/quality where it earns its cost.

## Playwright for agentic e2e
- Drive the dashboard through the real approval lifecycle; assert on stable roles/test-ids, not LLM-generated copy.
- Never assert exact model text — assert structure, presence of evidence/claim links, and that provenance renders. Use `expect(locator)` web-first auto-retrying assertions to absorb async agent latency.

## Checklist
- [ ] Pulled current docs (links above) — confirmed API names against today's docs, not memory.
- [ ] Each node testable as `state -> partial-state`; LLM/tool clients injectable.
- [ ] In-memory checkpointer + fixed `thread_id` in the eval harness.
- [ ] Dataset of representative releases with reference trajectories + final outputs.
- [ ] Trajectory eval chosen per node-criticality (strict for must-order policy steps, unordered/subset elsewhere) using the **graph** evaluators.
- [ ] Tool-call correctness eval (args + outcome).
- [ ] All 3 interrupt gates tested across approve/edit/reject with `Command(resume=...)`.
- [ ] pytest/vitest gate fails the build on threshold; cassettes committed.
- [ ] Cost + latency budgets asserted; Bedrock token/cost pricing configured in LangSmith.
- [ ] Playwright e2e asserts structure & provenance, not generated prose.
- [ ] Flaky-eval policy: judge variance bounded, retries/seeds documented, no silent score truncation.

## Gotchas
- **Bedrock breaks auto cost/token tracking.** LangSmith auto-records usage only for OpenAI/Anthropic/OpenAI-compatible. Bedrock Converse responses often arrive without recognized usage → **$0 cost, blank tokens**. Map `ls_provider`/`ls_model_name` + set custom per-1M pricing in workspace settings, or submit `usage_metadata` client-side. Don't gate on a cost metric that's silently zero.
- **`interrupt()` re-runs the whole node on resume.** Code *before* `interrupt()` executes again after `Command(resume=...)`. Side effects there (DB writes, S3 puts, Bedrock calls) double-fire and corrupt eval state. Put side effects after the interrupt or make them idempotent.
- **`__interrupt__` / `__start__` markers poison strict trajectory match.** Use `extract_langgraph_trajectory_from_thread` (which understands them) rather than diffing raw message lists; otherwise every HITL run fails strict mode for the wrong reason.
- **Missing checkpointer = no resume, no trajectory.** Without a checkpointer + `thread_id`, interrupts don't persist and trajectory extraction is empty. This is the #1 silent cause of "my eval returns nothing."
- **LLM-as-judge is itself nondeterministic.** A judge regression looks like an app regression. Pin/version the judge model+prompt, measure judge variance separately, and prefer deterministic `agentevals` matchers where a reference trajectory exists.
- **Uncached CI evals are slow and expensive and flaky.** Forgetting `LANGSMITH_TEST_CACHE` (cassettes) means every PR pays full token cost and inherits live-model variance. Commit cassettes; selectively cache hosts.
- **`--langsmith-output` hides real pytest errors** and is incompatible with `pytest-xdist`. When a gate fails inexplicably, drop the flag to see the actual traceback.
- **`log_inputs()` called twice overwrites** — silent data loss in a parameterized eval loop.
- **Asserting on generated copy in Playwright** = permanently flaky e2e. Model wording changes every run; assert DOM structure, test-ids, and evidence-link presence instead.
- **Swarm handoffs hide routing from traces.** `Command(goto=...)` jumps spread decision logic across tools, so trajectory evals can't see "who decided to hand off." If routing accuracy matters, use supervisor or add explicit handoff-correctness evals.
- **Reducer mismatch corrupts accumulated state.** Wrong `Annotated` reducer (overwrite vs. append) makes trajectories look right turn-by-turn but wrong in aggregate — unit-test reducers directly.

<!-- sources fetched at generation: https://docs.langchain.com/oss/python/langchain/test/evals, https://docs.langchain.com/langsmith/pytest, https://github.com/langchain-ai/agentevals, https://docs.langchain.com/langsmith/cost-tracking -->

From ichiorca/shipsignal, Apache-2.0.

More from ichiorca/shipsignal