debugging-experiments

Files (5)

• SKILL.md
• customer-reply.md · references/
• pulling-the-data.md · references/
• real-vs-noise.md · references/
• srm_check.py · scripts/

SKILL.md (first lines)

# Debugging experiments

PostHog Experiments are A/B tests: a feature flag randomizes users into variants, the SDK
records an **exposure** when the flag is read, and PostHog computes per-variant metrics and
significance. A customer looks at that results page and asks why it looks wrong.

**Most experiment-results tickets are config or exposure-collection problems, not statistics
bugs.** The randomization is fine; something upstream is skewing which users get exposed, or
stopping exposures from being recorded. The job is to find _which_, prove it with the
customer's own data, and hand back a plain-language explanation plus the fix.

This skill is the customer-support front door. It carries the two most common complaints
inline (uneven exposures, missing exposures) and loads
[`diagnosing-experiment-health`](../diagnosing-experiment-health/SKILL.md) as a diagnostic
library for the deeper long tail (interpretation traps, numbers-vs-SQL, mid-run surprises).

## Debugging workflow

1. **Parse the ticket.** Extract project ID, instance (US vs EU — the URLs and data live in
   different places), experiment ID or name, the `lib`/platform if relevant, the exact
   complaint in the customer's words, and what they already tried. Aged or multi-reply tickets
   are dirty: the config may have been edited mid-thread, so re-pull current state and treat
   earlier claims as stale.
2. **Resolve the experiment.** If the ticket names it rather than giving an ID, load
   [`finding-experiments`](../finding-experiments/SKILL.md) to resolve it, then call
   `posthog:experiment-get`.
3. **Pull the data read-only.** Run the fixed data-pull sequence in
   [references/pulling-the-data.md](references/pulling-the-data.md). This produces the
   "pertinent numbers" you will show the customer: per-variant exposed-person counts, `$multiple`
   share, the `distinct_id`/`person` fragmentation ratio, the SRM chi-squared result, the
   exposure trajectory, and the flag/experiment activity log. Verify from data before asking
   the customer anything.
4. **Match the complaint** to the known-cause catalog below. Confirm the single leading cause
   with one targeted number from step 3 before writing. Treat the customer's _own_ conclusion
   ("it's just noise", "a measurement bug") as a hypothesis to **disconfirm**, not confirm —
   pull the data independently rather than re-deriving their answer. Quantify a suspected cause
   before asserting its impact (count the contaminating cohort, don't eyeball it). One trap in
   particular: never run the SRM chi-square against an _assumed_ even split — read the configured
   `rollout_percentage` first, since an intended 34/33/33 reads as a ~2% SRM under an equal-split
   assumption.
5. **Scope the fix to the experiment's state** before recommending it. On a **draft**, config
   changes are free — recommend freely. On a **running** experiment every change has a mid-run
   tradeoff (changing the split is an anti-pattern — prefer reset or end+restart; see
   [`configuring-experiment-rollout`](../configuring-experiment-rollout/SKILL.md) and
   [`managing-experiment-lifecycle`](../managing-experiment-lifecycle/SKILL.md)). On a
   **stopped/shipped** experiment the flag and results are the documented outcome, so recommend
   interpretation or a _next_ experiment, not a mid-run edit. Don't propose reversing a state change
   unless the customer asks how to undo it.
6. **Write the reply** using [references/customer-reply.md](references/customer-reply.md):
   cause → fix → the numbers that prove it, in the customer's UI language.

## Known-cause catalog — "exposures aren't even" / "one variant has no traffic"

Ordered by how often they're the answer. Full mechanism detail lives in
[`diagnosing-experiment-health/references/bias-and-skew.md`](../diagnosing-experiment-health/references/bias-and-skew.md)
(group A) — load it when a case needs more depth than the summary here.

**First, split a real SRM into its two possible homes.** Assignment is a deterministic hash of a
stable identifier (the `distinct_id` by default; the device ID or group key for those flag types —
see [references/pulling-the-data.md](references/pulling-the-data.md)), so with an unchanged split
every user has a _fixed_ variant and any set of users must fall close to the configured percentages.
A confirmed SRM (chi-squared p < 0.001 at healthy volume — not eyeballed) therefore lives in exactly
one of two places:

- **Assignment-side** — the recorded variant disagrees with what the hash would assign. Something
  overrode assignment at serve time: a stale local-evaluation definition, an inherited bootstrap
  value, a forced release-condition variant, or a mid-run rehash.
- **Capture-side** — the recorded variant _agrees_ with the hash (assignment is fine), but _which_
  users get an exposure recorded is selected: one arm reaches a surface the other never does, or
  one arm's users read the flag before it loaded and are silently dropped.

The decisive test that tells you which half you're in — recompute the assignment hash offline, then
split the observed gap into the part explained by _which users got recorded_ (selection ⇒
capture-side) and the part explained by _users recorded onto the wrong arm_ (reassignment ⇒
assignment-side) — is the
[decisive test in references/pulling-the-data.md](references/pulling-the-data.md#the-decisive-test-recompute-assignment-offline),
with a runnable [`srm_check.py`](scripts/srm_check.py). Run it before guessing. It names a side only
when one component both dominates the gap and is statistically distinguishable from zero; otherwise
it reports the split as mixed, or the test as inapplicable, and says why. Don't route on the raw
agreement percentage — scattered disagreements can't produce a _directional_ SRM, so a large
capture-side skew under a little override noise still reads as high agreement. The causes below are
tagged with the half they sit in.

- **Uneven split + "Exclude from analysis" (the bias banner).** This is the most common real
  cause. When the variant split is uneven _and_ multiple-variant handling is set to **Exclude
  from analysis** (the default) _and_ some users were exposed to more than one variant, the
  excluded `$multiple` users are dropped asymmetrically — the smaller variant loses a larger
  fraction of its users, so it looks artificially worse. PostHog raises the **"Setup likely
  introduced bias"** banner once the `$multiple` share crosses 0.1%. Detect it purely from
  `posthog:experiment-get` (split + `exposure_criteria.multiple_variant_handling`) and the
  `$multiple` total from the exposure query. Fix: switch handling to **Use first seen
  variant**, and/or move to an even split.
- **Sample ratio mismatch (SRM).** The observed split is statistically far from the configured
  split. Confirm with the chi-squared test (p < 0.001) from
  [references/pulling-the-data.md](references/pulling-the-data.md) — don't eyeball ratios; a 2:1 skew
  at a few dozen exposures is normal noise. Count **people, not events** — run the test on the
  per-person `total_exposures` from `posthog:experiment-results-get`, since raw
  `$feature_flag_called` counts vary by how often each arm re-reads the flag and will manufacture an
  SRM that isn't there. Once confirmed, use the decisive test above to pick the half, then work the
  tagged causes below.
  Bot traffic and identity fragmentation are weak
  _directional_ causes — a crawler counts once per person, and fragmentation only inflates the
  excluded `$multiple` bucket — so suspect either only when it correlates with one arm.
- **Capture-by-surface (capture-side).** One arm reaches a page or screen the other never does, so
  it collects exposures the other structurally can't. Confirm: split the _first-exposure_ variant
  by `$pathname` / `$screen_name` (query in [references/pulling-the-data.md](references/pulling-the-data.md)).
  Some paths near 50% and others near 100% one variant ⇒ this is it; every path showing the same
  skew ⇒ capture-by-surface is out and the bias is upstream.
- **Flag read before it loaded (capture-side).** A user who evaluates the flag before flags have
  loaded (or who doesn't match a release condition) gets `false`/`undefined`, which the variant
  allow-list silently drops — so those users vanish from their arm instead of showing up wrong. If
  one arm is short by ~N persons, check whether the `false`/`null` person count (broken down by
  `$lib`/surface) is near N and concentrated on the short arm. If so, flag-read timing is the lead
  and the fix is in the customer's code.
- **Identity fragmentation.** The same person is split across multiple `distinct_id`s (usually
  `identify()` called _after_ the flag is read, or anonymous→identified transitions), so they
  appear in both arms and inflate the `$multiple` bucket (and, with an uneven split + Exclude,
  feed the bias banner above). Signal: `distinct_id`/`person` ratio noticeably above 1 (use 1.2 as
  a soft cue), or persons seen under more than one variant. On its own this does **not** create a

From PostHog/skills, MIT.