diagnosing-experiment-health

Files (8)

• SKILL.md
• bias-and-skew.md · references/
• diagnostic-snapshot.md · references/
• empty-experiment.md · references/
• interpretation.md · references/
• mid-run-changes.md · references/
• numbers-vs-sql.md · references/
• qualitative-feedback.md · references/

SKILL.md (first lines)

# Diagnosing experiment health

This skill answers: **My PostHog experiment results look wrong, biased, or empty — what's going on?**
It is the diagnostics and health check for one experiment: its setup, its exposures, its results and what changed during the run.

Match the user's complaint in the dispatch table, then read the matching reference file for the
diagnostic.

Each diagnostic in the reference files is tagged `[HIGH]`, `[MEDIUM]`, or `[LOW]` based on how
strongly it's verified — `[HIGH]` is verified directly in PostHog code, `[MEDIUM]` is partially or
team-source verified, `[LOW]` describes SDK/external behavior that wasn't verified here. Treat `[LOW]`
items as hypotheses to test, not facts to assert.

## Step 1 — Resolve the experiment

If the user refers to an experiment by name or description, load the `finding-experiments` skill first to
resolve it to a concrete ID.

Call `experiment-get` and pull these fields. They are inputs for almost every diagnostic:

- `status` (`draft` / `running` / `paused` / `exposure_frozen` / `stopped`), `start_date`, `end_date`, `is_legacy`
- `feature_flag.active`, and `feature_flag.filters.multivariate.variants[]` — the variant keys and the split (`rollout_percentage`).
  This is the flag as it is now. After a ship or an edit, the split of the run is in the change history (Step 1.5).
- `feature_flag.filters.aggregation_group_type_index` — when set, the experiment's unit is a group, not a person
- `feature_flag.filters.groups[]` — for each group read `variant`, `properties`, and
  `rollout_percentage` (that group's rollout, % of the matched bucketing units — persons, or groups
  when the flag or the group is aggregated by a group type — that enter the experiment; each
  group has its own, so there is no single overall rollout unless the flag has one unconditional
  group). A non-null `variant` that is one of the flag's variant keys is a forced-variant override on the matched cohort
  (release-condition assignment, not randomized) — surfaces A7. Watch for the severe shape (A7b): a
  variant-pinned group with broad/empty `properties` at high rollout, or no group left randomized
  (`variant: null`) / no release path to one arm — that starves the other variant (one arm gets ~0
  analyzable exposures). See `references/bias-and-skew.md`.
- `feature_flag.bucketing_identifier` and `feature_flag.ensure_experience_continuity` — what the flag hashes (A3, A6, A8, A10)
- `holdout` and `excluded_variants` — people and arms that are outside the analysis on purpose
- `exposure_criteria.multiple_variant_handling` — defaults to `"exclude"` if absent
- `exposure_criteria.exposure_config` — the exposure event and its property filters.
  Unset, or naming `$feature_flag_called`, means the default exposure event: read which one
  from `resolved_exposure_event`.
  When it is `$feature_flag_called`, also read `$experiment_exposure` (`event IN ('$feature_flag_called', '$experiment_exposure')`), and count persons, not rows.
  A config that names `$experiment_exposure` counts that event, whatever `resolved_exposure_event` says.
  Any other event, or an action, is a _custom exposure event_.
  Property filters in the config apply to either (B12).
  The table in `references/diagnostic-snapshot.md` says which property carries the variant.
- `exposure_criteria.activation_config` — an activation event, or an action, on top of the default exposure.
  When it is set, a person counts as exposed only from their first activation event at or after their first flag exposure, and the time of that event is the exposure time.
  It does not combine with a custom exposure event.
- `exposure_criteria.filterTestAccounts` — defaults to `true`
- `metrics` and `metrics_secondary`, with shared metrics — for each: the name, the type and the definition.
  Describe a metric from its definition, not from its name (D13).
- `stats_config` — `method` (Bayesian or Frequentist), the confidence level (`bayesian.ci_level`, `frequentist.alpha`), `frequentist.sequential_testing_enabled`, `cuped`, `baseline_variant_key`.
  A missing `method` reads as Bayesian and a missing confidence level as 95%. Sequential testing and CUPED fall back to the project default (C13).
- `running_time_calculation` — the target sample (a total across all variants), the days still missing to reach it, and the minimum detectable effect.
  When the estimate comes from numbers the user typed, or was written at creation and never refreshed by the page, the days are the whole estimated length (C12).
  A duplicated experiment starts with the values of its source until the page refreshes them.
- `only_count_matured_users` — E11

## Step 1.5 — Pull a diagnostic snapshot (verify before asking)

Before asking the user clarifying questions, pull the diagnostic snapshot in
[references/diagnostic-snapshot.md](references/diagnostic-snapshot.md). Most diagnostics in this skill
can be confirmed or ruled out from that data without an interview.

Read the cheap sources first, in every diagnosis: the stored results, then the change history.
When the user reports a surprising change, read the change history before the stored results.
Then the exposure data, when the diagnosis needs what only it holds.
Query raw events only for what those cannot show.

## Step 2 — Match symptom to diagnostic

| User says...                                                                               | Diagnostic group                             |
| ------------------------------------------------------------------------------------------ | -------------------------------------------- |
| "Smaller variant looks biased" / banner says bias                                          | A — bias & skew                              |
| "Variant ratio doesn't match my split" / mismatch warning / "is 58/42 normal?"             | A — bias & skew                              |
| "Why isn't it 50/50?" / "one variant gets fewer users every day"                           | A — bias & skew                              |
| "Users in both control and test" / high `$multiple` %                                      | A — bias & skew                              |
| "Our own analytics show an even split and PostHog doesn't"                                 | A — bias & skew                              |
| "Most responses on the flag are false" / "users saw the new UI for a second, then lost it" | A — bias & skew                              |
| Multi-variant exposure on a server-rendered app                                            | A — bias & skew                              |
| Banner about feature-flag/experiment state mismatch                                        | A — bias & skew                              |
| "Migrating distinct_id" / "switching from anonymous to user_id" mid-run                    | A — bias & skew                              |
| Metric count is much smaller than exposures (e.g. 10× or 100× gap)                         | A — bias & skew (route here before D)        |
| "Experiment shows 0 / not enough data" / empty                                             | B — empty experiment                         |
| "Variant always undefined / false"                                                         | B — empty experiment                         |
| "$feature_flag_called fires but no exposures show up"                                      | B — empty experiment                         |
| "Exposures dropped to zero after I added a filter" / "the copied experiment is empty"      | B — empty experiment                         |
| "Plenty of exposures, zero conversions"                                                    | B — empty experiment (then D11)              |
| "Experiment says running but exposures haven't moved in weeks/months"                      | B — empty experiment                         |
| "Significance keeps flipping as we run longer"                                             | C — interpretation traps                     |
| "Significance was declared, then it wasn't significant anymore"                            | C — interpretation traps                     |
| "It's been running one day, can we read it?"                                               | C — interpretation traps                     |
| "30/16 split at 46 exposures, is this broken?"                                             | C — interpretation traps                     |
| "It says complete / target reached and nothing is significant"                             | C — interpretation traps                     |
| "A/A test is showing significant results"                                                  | C — interpretation traps                     |
| "Many metrics — some significant, some not"                                                | C — interpretation traps                     |
| "The test variant does worse, is it set up correctly?" / "can you check this experiment?"  | Step 3 — the two checks, then C and D12      |
| "Bayesian says 96% chance to win — should we ship?"                                        | C — interpretation traps                     |
| "Confidence intervals overlap — does that mean not significant?"                           | C — interpretation traps                     |
| "The p-value is exactly 1" / "control shows a lift"                                        | C — interpretation traps                     |
| "An external tool (significance calculator or AI agent) disagrees with PostHog"            | C — interpretation traps                     |
| "Should I ship? Primary is up but a secondary is down"                                     | C — interpretation traps                     |
| "PostHog numbers ≠ my SQL count"                                                           | D — numbers vs SQL                           |
| "Funnel says X% but my raw event count says Y"                                             | D — numbers vs SQL                           |
| "The experiment says flat and our dashboard shows a drop"                                  | D — numbers vs SQL                           |
| "Sum of revenue looks wrong" / "breakdown shows 'none'"                                    | D — numbers vs SQL                           |
| "The metric's name says one thing and the number says another"                             | D — numbers vs SQL                           |
| "Recordings don't match the stats" / "no recordings for exposed users"                     | D — numbers vs SQL                           |
| "I applied a filter but the user count didn't change"                                      | D — numbers vs SQL                           |
| "I want to slice results by current person properties (as of now, not as of exposure)"     | D — numbers vs SQL                           |
| "The results haven't updated" / "refresh changes nothing"                                  | D — numbers vs SQL                           |
| "Changed split / rollout / metric / criteria mid-run, now odd"                             | E — mid-run changes                          |
| "The numbers changed overnight and nobody touched it"                                      | E — mid-run changes, then D8 in D            |
| "We paused it for a few days, can we still trust it?"                                      | E — mid-run changes                          |
| "Froze the experiment" / "after the freeze users lost the variant" / "freeze was refused"  | E — mid-run changes                          |
| "Ended/shipped — flag now flipped to 0/100 unexpectedly"                                   | E — mid-run changes                          |
| "We shipped the variant and some users still don't get it"                                 | E — mid-run changes                          |
| "Long-term metric moves opposite from primary"                                             | E — mid-run changes                          |
| "Retention metric counts users I didn't expect"                                            | E — mid-run changes                          |
| "Can't convert the feature flag back to a simple (boolean) flag after the experiment ends" | E — mid-run changes                          |

From PostHog/skills, MIT.