configuring-experiment-analytics

Files (4)

• SKILL.md
• interpreting-results.md · references/
• metric-configuration.md · references/
• metric-templates.md · references/

SKILL.md (first lines)

# Configuring experiment analytics

This skill answers: **Who is included in the analysis?** and **How to measure impact?**

## Exposure criteria

Exposure criteria determine which users are counted in the experiment analysis.

### Exposure event

Two options:

1. **Default exposure event** — users are included when the experiment's default exposure event fires for the experiment's flag: `$feature_flag_called`, or `$experiment_exposure` for newer experiments. Which one applies is resolved server-side — read `resolved_exposure_event` from `experiment-get` rather than assuming either name (both events carry the same properties). When it is `$feature_flag_called`, also read `$experiment_exposure` (`event IN ('$feature_flag_called', '$experiment_exposure')`), and count persons, not rows. This is the standard approach — it means a user is included only when they actually encounter the feature flag in your code.
2. **Custom exposure event** — users are included when a specific custom event fires. Use this when you want tighter control over who enters the analysis (e.g., only users who actually visit the page where the experiment runs).

### Multiple variant handling

When a user is exposed to multiple variants (e.g., due to flag changes or race conditions):

- **Exclude from analysis** — removes these users from the analysis entirely. Cleaner data, smaller sample.
- **Use first seen variant** — assigns users to the first variant they were exposed to. Keeps all users in the analysis. Note that "first seen" can introduce other biases as
  behavior cannot be clearly attributed to a single variant and is not recommended unless necessary.

**Bias risk on uneven splits.** "Exclude from analysis" combined with an uneven variant split can
introduce bias — multi-variant users are dropped asymmetrically and the smaller variant loses a larger
fraction of its assignments. If those users behave differently from the rest, the smaller variant's
metrics will be skewed.

The right mitigation depends on experiment state:

- **Not yet launched, or only exposed to a few users so far** — switch to an even variant split and
  use the overall rollout percentage to limit test-variant exposure. This removes the bias and
  preserves statistical power. See `configuring-experiment-rollout`.
- **Live experiment with significant exposures** — changing the split mid-run reassigns users across
  variants, which is bad for user experience and data quality. Switch this setting to "Use first
  seen variant" instead — it keeps already-assigned users in their original variant (no reassignment) and
  removes the asymmetric exclusion.

### Filter test accounts

`exposure_criteria.filterTestAccounts` (default: true) — excludes internal/test users from the analysis.

## Resolving experiments

Metric changes require an experiment ID. If the user refers to an experiment by name
or description (e.g. "add metrics to the checkout test"), load the `finding-experiments`
skill to resolve it to a concrete ID before proceeding.

## Metrics

A metric reaches an experiment one of two ways, both via `experiment-update`:

- **Inline metric** — defined directly on the experiment. Sent in the `metrics` array, which
  **replaces** the entire inline list, so always get the current experiment first via `experiment-get`
  to preserve existing metrics.
- **Shared (saved) metric** — a reusable metric object that can be attached to many experiments.
  Attached by ID via `saved_metrics_ids` (this list also **replaces** the experiment's existing
  saved-metric links, so resend the full set — see Step 1).

**Prefer reusing a shared metric over duplicating it inline.** Build a new inline metric only when
no suitable shared metric already exists.

### Step 1: Check for an existing shared metric (REQUIRED — match by definition, not name)

Before building any new inline metric, you MUST check whether the project already has a shared
(saved) metric that measures the same thing, and reuse it. Duplicating a metric that already exists
as a shared metric fragments measurement and is exactly what we want to avoid.

**Reuse is decided by the metric _definition_ — the event or action plus the metric type — not the
name.** Saved metrics are named by each team's own conventions, which you cannot guess, so you must
compare on what each metric measures (its `query`), never on its title.

**Workflow:**

1. **Know what you're about to build first.** Settle the target event(s)/action(s) and metric type
   (mean / funnel / ratio / retention) before searching — see Step 2 to confirm the event exists via
   `read-data-schema`. You can only recognize a duplicate once you know the concrete event/action,
   so this check runs _after_ you've pinned down the event, not before.
2. **Search by the event, then compare each candidate's `query`.** Call `experiment-saved-metrics-list`
   with `?event=<the event you're measuring>` to find metrics that reference it — matched directly (an
   `EventsNode`) **or** via the step events of any action a metric references, so action-based metrics are
   found by the event their action fires on. Then for each returned row, inspect its **`query`** (not the
   `name`/`description`): a saved metric is a reuse match when its `query` measures the **same event or
   action with the same `metric_type`** (and compatible `math`) as the metric you'd otherwise build, even
   if its name is different.
   - **Match on the event, not the action's name.** An action-based metric is discoverable by the event
     the action fires on — pass that event, not the action's label.
   - **Do not use `search` for this.** `search` matches only the metric's own `name` / `description` / tags —
     never the underlying event or action — so it cannot find a definition match. Use `search` only when the
     user names a specific saved metric to attach (name resolution, not a definition match).
3. **If a saved metric matches the definition** — confirm the match with the user by name/description,
   then attach it instead of building a new one:
   - Call `experiment-get` to read the experiment's current `saved_metrics`.
   - Call `experiment-update` with `saved_metrics_ids` set to the full desired set — it **replaces**
     existing links, so include the already-attached ones plus the new entry. Each entry has shape
     `{ "id": <saved-metric id>, "metadata": { "type": "primary" } }` — set `type` to `"primary"` or
     `"secondary"`. `metadata` is optional and defaults to primary.
   - **Watch the id when rebuilding the set:** each item in the `saved_metrics` you just read has a
     top-level `id` (the _link_ id) AND a `saved_metric` field (the _metric_ id). `saved_metrics_ids`
     wants the **`saved_metric`** value, not the link `id` — sending the link `id` attaches the wrong
     metric or fails validation.
   - You do not need to build the inline metric — the shared metric already encodes its events.
4. **If nothing in the library measures the same event/action + type** — build an inline metric
   (Step 2+). When that inline metric is likely to be reused across experiments, offer to create it
   as a shared metric instead, via `experiment-saved-metrics-create`, then attach it as above, so the
   next experiment can reuse it.

### Step 2: Discover available events (REQUIRED before building an inline metric)

Before suggesting or building any new inline metric, you MUST call `read-data-schema` to discover
what events actually exist in the project. Do NOT skip this step. Do NOT suggest event names
based on what you think the project might track — only use events you have confirmed exist.
(Attaching an existing shared metric from Step 1 does not need this — it already encodes its events.)

This applies even when:

- The user provides event names — look them up to confirm they exist and are spelled correctly
- The user asks "what metrics do you suggest?" — look up events first, then suggest from real data
- The context makes certain events seem obvious — they may not exist or may be named differently

From PostHog/skills, MIT.