creating-replay-vision-scanners

Files (1)

• SKILL.md

SKILL.md (first lines)

# Creating Replay Vision scanners

A scanner is a standing LLM probe over session recordings. Once created and enabled, it runs on a
**Temporal schedule that sweeps every 5 minutes**, applying its prompt to each new matching recording and
recording the result as an observation (a queryable `$recording_observed` event). Each observation spends
**credits** (1 credit = $0.01) from the org's budget for the current billing period, and an observation's
price depends on the scanner's `model` — so size spend in credits, and convert any observation count a user gives you into credits at that model's price.

That schedule is exactly why creation needs a gut-check: a scanner with a permissive query and full sampling
starts spending automatically and can drain the whole period's budget within its first few sweeps.
Creation itself does **not** check quota — that protection only kicks in at observation time, by which point
the budget may already be gone.

## First: is a scanner even the right thing?

A scanner is a **standing watch over future recordings**. If the user has specific sessions in front of them
and a question about those sessions, they don't want a scanner at all — they want `vision-scanners-inline-scan`,
which takes `session_ids` plus a `prompt`, saves nothing, and schedules nothing.

Use an inline scan when the sessions are already known: "what went wrong in these five recordings", "did any
of yesterday's checkout sessions hit the coupon bug", anything you'd otherwise answer by creating a scanner
and deleting it afterwards. It costs the same credits per session and reuses answers when the same question
is asked twice, so re-asking is cheap.

Create a scanner only when the user wants recordings that **haven't happened yet** to be scanned automatically.
If you find yourself planning to create a scanner, read its results once, and delete it, stop and run an
inline scan instead — a throwaway scanner leaves a scheduled sweep running against every future recording that
matches its query.

## Core principle: size before you ship

Never create an enabled scanner blind. Estimate its monthly credit spend, check the remaining credit budget,
and — when the projected spend is a meaningful fraction of what's left — show the user the numbers and get
confirmation before creating. This is the heart of the skill; the rest is supporting detail.

## The flow

### Step 1: What should the scanner do?

If the question, type, or prompt is still vague, run [[designing-replay-vision-scanners]] first. It turns a wish
into one visible question with a fixed answer shape, a proof rule, and permission to say inconclusive. Come back
here once those are settled.

Pick a `scanner_type` and write its `scanner_config`. Every type needs a `prompt`; the rest is type-specific:

| Type         | What it produces                                                  | `scanner_config` shape                                                                                                                                                      |
| ------------ | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `monitor`    | Open-ended observation against a prompt (e.g. "flag rage clicks") | `{"prompt": "..."}`; optional `"allow_inconclusive": true` (off by default, so the model must answer yes or no)                                                             |
| `classifier` | Assigns tags from a fixed label set                               | `{"prompt": "...", "tags": ["tag-a", "tag-b"]}` — `tags` needs ≥1 entry; optional `"multi_label": false` (defaults to true), `"allow_freeform_tags": true` (off by default) |
| `scorer`     | Numeric score on a rubric                                         | `{"prompt": "...", "scale": {"min": 1, "max": 5, "label": "frustration"}}` — `min` < `max`; `label` optional                                                                |
| `summarizer` | Free-text summary                                                 | `{"prompt": "..."}`; optional `"length": "short" \| "medium" \| "long"` (default `"medium"`). Embeddings are always on                                                      |

`scanner_type` is **locked after creation** — to change it you delete and recreate, so confirm the type is
right up front, and get the `scanner_config` shape right (a wrong shape is a create error, not a silent
default — unknown keys are rejected too).

If the user's intent makes the type and prompt obvious, just proceed — don't interrogate them.
When they only have a goal ("find where users get stuck in onboarding"), `vision-scanners-draft` drafts the type, config, query, model and sampling for you to review; it saves nothing.
Ask the user how much it should use, and offer both choices: free (`free_only: true`, runs only on the organization's free monthly credits and never adds to the bill) or a monthly credit budget they name. Never invent a budget. Pass the draft's `free_only` and `credit_limit` on to `vision-scanners-create`.

### Step 2: Which sessions?

The `query` is a `RecordingsQuery` shape that selects which recordings the scanner watches. `date_from` and
`date_to` are **ignored** (the schedule controls time), so don't bother setting them. Narrow the query to the
sessions that actually matter — by event, URL, person property, duration, etc. A narrow query is the single
biggest lever on cost.

When the target is one experiment's exposed population, that's its own job — use the
`scanning-experiments-with-replay-vision` skill, which derives this query from the experiment's exposure
criteria instead of hand-building it.

Two levers narrow it further, applied in this order:

- `sampling_mode` (default `comprehensive`) is a quality pre-filter on the matched sessions: `focused` keeps
  only the top sessions by surfacing score, `balanced` drops the lowest-quality ones, `comprehensive` keeps
  everything. Use it to spend the budget on sessions worth watching rather than shrinking coverage at random.
- `sampling_rate` (0..1, default 1.0) is a random downsample applied after that. Lower it to trade coverage
  for budget. Exactly 0 pauses scanning; non-zero rates below 0.0001 are rejected.

#### Which model?

`model` sets the price of every observation the scanner makes, so it's a cost lever as much as a quality one:
`gemini-3.5-flash-lite` (2 credits), `gemini-3-flash-preview` (5 credits, the default) and `gemini-3.8-flash`
(15 credits). Start at the default and only reach for `gemini-3.8-flash` when the cheaper tiers demonstrably
miss what the scanner is looking for.

### Step 3: Size it — the gut-check (do not skip)

Before creating, run both checks and reason about them together:

1. **Estimate spend** — call `vision-scanners-estimate` with the proposed `query`, `sampling_rate`,
   `sampling_mode` and `model`. It returns `matched_sessions_in_window`, the `window_days` measured,
   `estimated_observations_per_month`, `credits_per_observation`, `estimated_credits_per_month`, and
   `other_enabled_scanners_monthly_credits` (what the org's other enabled scanners are already projected to
   spend). When editing an existing scanner, pass its `scanner_id` so its own estimate isn't counted twice.
2. **Check budget** — call `vision-quota-get` for `remaining` and `exhausted` against the org's
   `credit_limit` (credits, 1 credit = $0.01; `null` when uncapped), plus the `period_start`/`period_end`
   of the current period.

Compare credits with credits over the same horizon, the way the product UI does — `remaining` is denominated
in credits, not observations, so comparing it against `estimated_observations_per_month` understates the cost
by the model's per-observation price.
`remaining` is what's left for the rest of the current period, so prorate the monthly projection to that
window rather than comparing a full month against it:

```text
fleet_monthly    = estimated_credits_per_month + other_enabled_scanners_monthly_credits
period_days      = period_end - period_start        (in days)
days_left        = period_end - now                 (in days, floored at 0)
rest_of_period   = fleet_monthly * days_left / period_days
```

Then decide on `rest_of_period` against `remaining`:

- If it comfortably fits within `remaining`, proceed.
- If it's a large fraction of (or exceeds) `remaining`, **stop and tell the user the concrete numbers**
  (e.g. "This scanner is projected to spend ~X credits/month, about $Y; over the N days left this period
  that's ~R credits against the Z you have left."), then confirm before creating. Tightening the `query`,
  switching `sampling_mode` to `focused`, lowering `sampling_rate`, or picking a cheaper `model` are all

From PostHog/skills, MIT.