analyzing-experiment-session-replays

Files (1)

• SKILL.md

SKILL.md (first lines)

# Analyzing experiment session replays

This skill guides you through analyzing session recordings for experiment variants to understand behavioral differences between control and test groups.

## When to use this skill

Use this skill when:

- The user asks to analyze session replays for an experiment
- The user wants to understand how users behave differently across experiment variants
- The user asks to compare user behavior between control and test variants
- The user wants qualitative insights to complement experiment metrics
- The user asks questions like "How are users behaving in my experiment?" or "Show me session replays for variant X"

## Prerequisites

Before analyzing session replays:

1. The experiment must be **launched** (not in draft state)
2. Session replay must be enabled for the project
3. Users must have been exposed to the experiment variants
4. The experiment must have a start date

## Workflow

### 1. Get experiment details and feature flag variants

First, retrieve the experiment information and the feature flag variants (source of truth).

**Step 1a: Get experiment metadata**

You can either:

- **Option A**: Use the `experiment-get` tool if you already have the experiment ID from context
- **Option B**: Query the experiments table via HogQL:

```sql
SELECT
    e.id,
    e.name,
    f.key AS feature_flag_key,
    e.start_date,
    e.end_date
FROM system.experiments e
JOIN system.feature_flags f ON f.id = e.feature_flag_id
WHERE e.id = <experiment_id>
```

From the experiment data, extract:

- `feature_flag_key`: The feature flag controlling the experiment
- `start_date` and `end_date`: The experiment's time range

**Step 1b: Get variants from the feature flag**

**IMPORTANT**: Always get variants from the feature flag, NOT from `experiment.parameters.feature_flag_variants`.
The parameters can be out of sync or deprecated. The feature flag is the source of truth.

Query the feature flag to get the current variants:

```sql
SELECT filters.multivariate.variants AS variants
FROM system.feature_flags
WHERE key = '<feature_flag_key>'
```

Select the variants path directly — selecting the whole `filters` object gets truncated in results for flags with large targeting configs.
Example structure: `[{"key": "control", "name": "Control", "rollout_percentage": 50}, {"key": "test", ...}]`

The variant `key` values (e.g., "control", "test", "variant_a") are what you'll use to filter session recordings.

### 2. Build session recording filters for each variant

For each variant in the experiment, construct recording filters that match users exposed to that variant.

**Filter structure for a variant** (input to `query-session-recordings-list`):

```json
{
  "date_from": "<experiment.start_date>",
  "date_to": "<experiment.end_date or current time>",
  "filter_test_accounts": true,
  "properties": [
    {
      "type": "event",
      "key": "$feature/<feature_flag_key>",
      "operator": "exact",
      "value": ["<variant_key>"]
    }
  ]
}
```

**Key points:**

- The `$feature/<flag_key>` event property records the flag's value on each event — filtering on it matches recordings where the flag was active with that variant. This is an approximation of exposure, broader than the experiment's exposure event (`$feature_flag_called`, or `$experiment_exposure` on the new rollout — both deduped per identity): right for browsing behavior across variants, but not an exact mirror of the analysis population — the `scanning-experiments-with-replay-vision` skill derives that exact filter when you need it
- `value` is an array of variant key strings (e.g. `["control"]`); for boolean flags use `["true"]` or `["false"]`
- Avoid the `type: "flag"` / `flag_evaluates_to` property filter for variant scoping — the recordings query accepts it but silently ignores it, returning unfiltered results (last verified 2026-06-10). If you want to try it anyway, verify it actually filters first: a query with a nonexistent flag key should return zero recordings
- Set the date range to the experiment's start and end dates
- Enable `filter_test_accounts: true` to exclude test users

### 3. Retrieve recordings for each variant

Use the `query-session-recordings-list` tool with the filters constructed in step 2.

Call the tool once per variant to get recordings for each group:

- Variant "control" → recordings for control group
- Variant "test" → recordings for test variant
- Additional variants if the experiment has more than 2

The tool returns a list of recordings with metadata including:

- `distinct_id` — the person's distinct ID
- `recording_duration`, `active_seconds`, `inactive_seconds`
- `click_count`, `keypress_count`, `mouse_activity_count`
- `console_log_count`, `console_warn_count`, `console_error_count`
- `start_url` — first page URL visited
- `start_time` / `end_time`, `activity_score`

From PostHog/skills, MIT.