a-b-testing

Files (4)

• SKILL.md
• framework-notes.md · references/
• check-output.py · scripts/
• output-template.md · templates/

SKILL.md (first lines)

# A/B Testing

## Overview

A/B Testing brings experimental rigor to GTM optimization. The core principle: most "optimizations" are guesswork dressed as strategy. Teams change subject lines because someone "feels" they're better, rewrite copy based on a single positive reply, or abandon channels because of one bad week. This skill prevents the waste of optimizing based on noise rather than signal.

The non-obvious rule: statistical significance is necessary but insufficient. A statistically significant 10% improvement in open rate that produces 0% more meetings is a distraction. Optimize for downstream revenue metrics — reply rate, meeting rate, conversion rate — not vanity metrics. Every test must trace its impact through the full funnel.

This skill produces: an Experiment Design Document with hypothesis, success metrics, power analysis, and methodology; a prioritized ICE/PIE test backlog; a 30/60/90-day testing roadmap; and a Results Analysis Report with statistical tests, confidence intervals, and scale/stop/kill recommendations.

## When to Use

- User says "A/B test" or "split test" → activate this skill
- User mentions "test subject lines" or "test email copy" → use for experimental design
- User asks "is this statistically significant" or "what sample size do I need" → activate for statistical guidance
- User says "experiment design" or "test plan" → use this skill
- User asks "how do I prioritize tests" or "what should I test next" → use ICE/PIE scoring
- User mentions "testing roadmap" or "experiment cadence" → plan the roadmap
- Trigger phrases: A/B testing, split testing, experiment, statistical significance, confidence interval, sample size, test variant, control group, multivariate test

Do NOT use for:
- Analyzing existing campaign performance without test design → use campaign-analytics
- Writing copy variants → use cold-email-copywriting (run after this skill for variant creation)
- General campaign analytics and benchmarking → use campaign-analytics
- Setting up sequences or cadences → use cold-email-strategy
- Landing page CRO (broader scope) → use landing-pages

## Authoritative Foundations

This skill draws from the following established methodologies:

- **Sean Ellis (GrowthHackers) — Experimentation System** — High-tempo testing framework: at least one test per week, backlog-driven, hypothesis-first. Ellis's work at Dropbox, Eventbrite, and LogMeIn established the growth experimentation discipline. The core principle: testing velocity matters more than test size. Two tests per week with 100 sample size beats one test per month with 10,000.

- **GrowthHackers ICE Scoring Model** — Impact, Confidence, Ease framework for test prioritization. Each test idea scored 1-10 on three dimensions. ICE ensures you test the most valuable ideas first, not the most interesting ones.

- **PIE Framework** (WiderFunnel/Chris Goward) — Potential, Importance, Ease. Alternative to ICE for companies further along the optimization maturity curve. PIE weights strategic importance higher than ICE which weights ease.

- **Optimizely / VWO — Statistical Engine** — Sequential testing, false discovery rate control, and multi-arm bandit approaches. Modern experimentation platforms have moved beyond simple t-tests. This skill applies these principles without requiring those specific platforms.

- **Ron Kohavi (Microsoft/Airbnb) — Trustworthy Online Controlled Experiments** — The definitive academic work on A/B testing at scale. Kohavi's rules: never stop a test early based on "significance," always pre-register hypotheses and metrics, segment results by new vs returning users.

## Prerequisites

- Historical campaign data with per-variant metrics: sent, delivered, opened, clicked, replied, meetings booked per variant (minimum 200 sends per variant for significance)
- Campaign platform that supports variant splitting (Smartlead, Instantly, Apollo, or manual list splitting)
- Baseline metrics to compare against: current open rate, reply rate, meeting rate for the campaign being tested
- Optional: statistical software or spreadsheet with CHISQ.TEST, T.TEST functions (Google Sheets, Excel, or Python/R)
- Recommended upstream skill: campaign-analytics (to identify which metrics to test)

## Step-by-Step Process

### Phase 1: Intake

Gather required information from the user. Ask all questions at once. Do not proceed until all answers are received.

Required intake questions:

1. **What are you testing?** Subject line, email body copy, CTA, send time, sender name, personalization level, sequence structure, channel, offer, landing page, or other.
2. **What is the current baseline?** Current metric value for the element being tested (e.g., current reply rate is 2.1%).
3. **What is the minimum detectable effect?** What improvement would be practically meaningful? (e.g., "We want to detect a 30% relative improvement from 2.1% to 2.7%")
4. **What is your primary success metric?** What metric determines if the test wins? Must be a downstream metric (reply rate, meeting rate, conversion rate), not a vanity metric (open rate).
5. **What is your current sending volume?** How many emails per day/week can you allocate to this test?
6. **What is the test duration constraint?** How long can the test run? (1 week? 2 weeks? 4 weeks?)
7. **What are the risk constraints?** Is there a maximum acceptable performance drop? (e.g., "We can tolerate up to a 20% decrease in reply rate")

### Phase 2: Experiment Design

Design the experiment with statistical rigor:

1. **Formulate the hypothesis** in if-then-because format:
   - IF we [change specific element], THEN [metric] will [direction] by [magnitude], BECAUSE [behavioral rationale].
   - Example: "IF we replace the direct meeting ask CTA with a low-friction 'worth a conversation?' question, THEN reply rate will increase by 30%, BECAUSE recipients perceive lower commitment in responding to a question versus scheduling a meeting."

2. **Define primary and guardrail metrics**:
   - **Primary metric**: The metric you're trying to improve (must be downstream: reply rate, meeting rate, conversion).
   - **Guardrail metrics**: Metrics that must not degrade significantly (e.g., unsubscribe rate must stay <0.5%, spam complaint rate must stay <0.1%, positive reply sentiment must stay >70% of replies).
   - **Vanity check metrics**: Open rate, click rate — tracked but not used for decisions unless they trigger guardrail violations.

3. **Power analysis — determine required sample size**:
   ```
   For proportions (reply rate, open rate, conversion rate):
   n = (Z_α/2 + Z_β)² × (p₁(1-p₁) + p₂(1-p₂)) ÷ (p₁ - p₂)²

   Where:
   - Z_α/2 = 1.96 (for 95% confidence, two-tailed)
   - Z_β = 0.84 (for 80% power)
   - p₁ = baseline proportion (e.g., 0.021 for 2.1% reply rate)
   - p₂ = expected proportion after improvement (e.g., 0.027 for 2.7%)
   ```

   Simplified approximation for common GTM test scenarios:

   | Baseline Rate | Minimum Detectable Effect (MDE) | Required Sample Per Variant |
   |---------------|-------------------------------|----------------------------|
   | 1% | 50% relative (to 1.5%) | 8,500+ |
   | 1% | 100% relative (to 2.0%) | 2,800+ |
   | 2% | 50% relative (to 3.0%) | 4,600+ |
   | 2% | 100% relative (to 4.0%) | 1,300+ |
   | 5% | 50% relative (to 7.5%) | 2,000+ |
   | 5% | 100% relative (to 10%) | 500+ |
   | 10% | 50% relative (to 15%) | 1,100+ |
   | 10% | 100% relative (to 20%) | 250+ |

   **Critical rule**: Never run a test with fewer than 200 sends per variant. Below 200, even large effects are indistinguishable from random noise.

4. **If sample size cannot be met within time constraints**:
   - Option A: Accept larger MDE (test for bigger improvements only)
   - Option B: Reduce variants (2 variants not 4 minimizes sample requirement)
   - Option C: Extend test duration
   - Option D: Don't run the test. Making decisions on insufficient data is worse than not testing.

5. **Design the variant split**:
   - **Control**: Current version. Always include a control. Testing Variant A vs Variant B without a control tells you which is relatively better, not whether either is an actual improvement.
   - **Variant A/B/C**: Test variants. Limit to 2-3 variants per test. More variants require more sample and slower learning.
   - **Randomization**: Random split (not alternating, not time-based). Ensure each contact has equal probability of receiving each variant.
   - **Segmentation**: If testing across segments (ICP tiers, personas, channels), ensure each segment has adequate sample size independently.

6. **Define stopping rules**:
   - **Do NOT peek**: Do not check results and stop early when they "look significant." Early stopping inflates false positive rates from 5% to 25-30%. Set a fixed end date.
   - **Exception — harm detection**: If a variant's guardrail metrics trigger (e.g., unsubscribe rate spikes above 1%), stop that variant immediately.

From LeadMagic/gtm-skills, MIT.

More from LeadMagic/gtm-skills