creating-online-evaluations

Files (2)

• SKILL.md
• evaluation-payload.md · references/

SKILL.md (first lines)

# Creating online evaluations

An **online evaluation** automatically scores either each matching `$ai_generation` or the whole trace
containing it, until disabled. A good eval comes from a real failure mode you've found in production traffic,
not from a guess or a generic metric like "hallucination" or "helpfulness". This skill starts once those
failure modes are identified and turns them into scoped, continuously-running evals.

**One eval per failure mode, and as many evals as the data justifies.** How many to create is a judgment
call you make from what the traces actually showed — sometimes one, often three or four. Never assume the
answer is one, and never bundle unrelated modes into one evaluator. The exception is a set of modes that are
outcomes of the same check, such as the reasons a reply missed the user's request: one categorical eval with
a category per mode fits that (2.1).

**Propose before you create.** Bring the user a short list of candidate evals and let them pick which ones
they want (Phase 1.1). Creating evals they didn't ask for costs them money and noise.

**First, know what you're evaluating.** Finding and ranking the failure modes worth catching is a
separate job. If the user doesn't specify what they want to evaluate, ask them. If they are still vague
about it and don't refer to a specific failure mode, run `exploring-ai-failures` to scope a use case,
find failing traces, and produce a ranked list of failure modes.

For the mechanics of _writing and iterating_ an evaluator (Hog source vs LLM-judge prompt, dry-running,
debugging a live eval), defer to `exploring-llm-evaluations`.

## Tools

| Tool                                        | Purpose                                                       |
| ------------------------------------------- | ------------------------------------------------------------- |
| `posthog:llma-evaluation-config-get`        | Check the active provider key used by unpinned judges         |
| `posthog:llma-provider-key-list`            | Find a usable (`ok` state) provider key to pin                |
| `posthog:llma-evaluation-judge-models`      | List valid provider+model combos                              |
| `posthog:llma-evaluation-directory-list`    | List directories available for organizing the evaluation      |
| `posthog:llma-evaluation-directory-create`  | Create a directory when the user asks for a new one           |
| `posthog:llma-evaluation-test-hog`          | Dry-run Hog source against recent generations before creating |
| `posthog:llma-evaluation-create`            | Create the evaluation (always `enabled: false` first)         |
| `posthog:llma-evaluation-run`               | Spot-run a draft eval against one generation                  |
| `posthog:llma-evaluation-update`            | Iterate config, then flip `enabled: true`                     |
| `posthog:llma-evaluation-backfill-estimate` | Count what a backfill over past data would evaluate           |
| `posthog:llma-evaluation-backfill-create`   | Evaluate past data, only after the user confirms the estimate |
| `posthog:llma-evaluation-backfill-get`      | Track a backfill's progress and coverage                      |
| `posthog:llma-evaluation-backfill-list`     | Find an evaluation's backfills, including one already running |
| `posthog:llma-evaluation-backfill-cancel`   | Stop a running backfill when the user asks                    |
| `posthog:execute-sql`                       | Verify a condition matches the events and volume you expect   |
| `posthog:generate-app-url`                  | Build a region- and project-qualified deep link to the eval   |

The full create payload (every field, the config schemas, the exact `conditions` shape) is in
[references/evaluation-payload.md](references/evaluation-payload.md).

## Phase 1 — Decide what to propose, then let the user choose

Start from real, observed failures, not metrics you picked in advance. If you don't already have them,
run `exploring-ai-failures` to scope a use case, find failing traces, and produce a ranked list of failure
modes — then come back.

### 1.1 — Turn the failure modes into a candidate set

**Let the data decide how many.** One failure mode is one eval, so a ranked list of four distinct modes is
a candidate set of four evals. Don't collapse them into one evaluator that tries to catch everything, and
don't stop at the top mode when the traces clearly showed more worth watching. Keep a candidate when:

- **It hurts.** Frequent or painful. A handful of modes usually covers the majority of failures.
- **It's checkable.** It reduces to one crisp criterion — "the reply must stay on the user's topic", "the
  tool call must include an `order_id`". If you can't state it in a line, it isn't ready to propose.
- **It's distinct.** Two candidates that would fail on the same generations are one eval.

Rank by how much they hurt and propose roughly the top five; mention in a line that you set weaker ones
aside rather than silently dropping them.

### 1.2 — Propose the set in plain language

**Never create evals the user hasn't picked.** Lay out the candidates and ask which ones they want.

Assume they haven't read the traces with you and don't know the eval vocabulary. Keep each one to a line or
two — a wall of text per eval means they can't compare them — but make it obvious what it watches and what
they'll see when it fails. No Hog snippets, property filters, or `hog`/`llm_judge` internals here. Number
them so they can reply "1 and 3":

> Found 3 failure patterns worth watching. Which should I set up?
>
> **1. Replies drift off topic** — checks the answer addresses what the user actually asked.
> Catches support replies that confidently answer a different question. Seen ~40×/day.
>
> **2. Order lookups missing an order ID** — checks every `lookup_order` call includes an `order_id`.
> Catches the silent tool failures that make the agent invent an order status. Seen ~15×/day.
>
> **3. Refuses questions it can answer** — checks the reply isn't a needless "I can't help with that".
> Catches users being turned away from things the docs cover. Costs an LLM call per check.

Per eval: what it checks, what bad thing it surfaces, and rough volume when you know it (Phase 2.5 verifies
it properly). Flag the ones needing an LLM judge, since those cost per run — that's the only mechanic worth
exposing up front. Keep the ranking implicit in the order; skip scoring tables.

Recommend a starting point if they seem unsure (usually the top one or two), and treat "all of them" or "go
ahead" as accepting the whole set. If a prompt tweak would likely fix a mode, suggest the fix alongside the
eval rather than in place of it — a rising pass rate is how they confirm the fix landed.

### 1.3 — When nothing surfaced, propose sentiment

If the traces were read and no failure mode is worth an eval, don't invent a failure and don't come back
empty-handed. Say what you looked at, then propose a `sentiment` eval as the floor:

> No clear failure pattern in the last 7 days. Worth starting with:
>
> **1. User frustration** — labels each user message positive, neutral, or negative.
> Shows which conversations are going badly, which is usually where the real failures hide. No judge cost.

It needs no provider key, it's cheap, and it gives them a signal to come back to once there's enough traffic
to spot patterns.

**No generations means no eval of any kind.** Sentiment only scores matching `$ai_generation` events, so if
the project has none, every eval you could create — sentiment included — would sit there never firing. Don't
propose one. Point them at `instrument-llm-analytics` to get AI observability capturing generations first,
and come back to this skill once there's traffic to read.

## Phase 2 — Build each accepted eval

Run 2.1 through 2.5 once per eval the user picked, so each one lands as a verified draft. **Leave every one
of them disabled until the whole set is verified** — 2.6 is a single pass over the finished set at the end,
not the last step of each loop. Enabling eval 1 while eval 3 is still being written puts a partially live

From PostHog/skills, MIT.