ab-testing

Files (8)

• SKILL.md
• ADMIN-DASHBOARD.md · references/
• GA4-GTM-TRACKING.md · references/
• PERSONALIZATION.md · references/
• QUICK-REFERENCE.md · references/
• RUST-WASM.md · references/
• STATISTICAL-ANALYSIS.md · references/
• VARIANT-ASSIGNMENT.md · references/

SKILL.md (first lines)

# A/B Testing Platform for SaaS

> **Stack**: Next.js 16 + Edge Middleware + GA4/GTM + Supabase + Rust-WASM (for stats)
> **Why DIY?**: Google Optimize sunset Sept 2023. GA4 has no native A/B testing.

## Quick Start

```tsx
// middleware.ts - Server-side assignment (no flicker)
import { NextResponse, type NextRequest } from 'next/server'

export function middleware(request: NextRequest) {
  const response = NextResponse.next()

  if (!request.cookies.get('exp_hero')) {
    const variant = Math.random() < 0.5 ? 'A' : 'B'
    response.cookies.set('exp_hero', variant, { maxAge: 60*60*24*30, path: '/' })
  }

  return response
}
```

```tsx
// Track with GA4
window.dataLayer?.push({
  event: 'experiment_view',
  experiment_name: 'hero_test',
  experiment_variant: variant
})
```

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                         A/B TESTING FLOW                            │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   1. ASSIGN (Edge Middleware)                                       │
│   ══════════════════════════                                        │
│   Request → Check cookie → Random assign → Set cookie → Response    │
│   ✓ No flicker (server-side)  ✓ Consistent (cookie-based)          │
│                                                                     │
│   2. RENDER                                                         │
│   ═════════                                                         │
│   Server/Client Component → Read cookie → Show variant              │
│                                                                     │
│   3. TRACK (GTM + GA4)                                              │
│   ════════════════════                                              │
│   dataLayer.push → GTM triggers → GA4 events with variant param     │
│                                                                     │
│   4. ANALYZE                                                        │
│   ═════════                                                         │
│   GA4 Explorations OR Supabase + Rust-WASM Bayesian analysis        │
│                                                                     │
│   5. PERSONALIZE (Advanced)                                         │
│   ═════════════════════════                                         │
│   Contextual bandit → Best variant per user segment                 │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Workflow

- [ ] 1. Define experiment config (name, variants, weight, segment rules)
- [ ] 2. Implement middleware assignment (see [VARIANT-ASSIGNMENT.md](references/VARIANT-ASSIGNMENT.md))
- [ ] 3. Wire up GTM tracking (experiment_view, conversions with variant param)
- [ ] 4. Register `experiment_variant` as Custom Dimension in GA4
- [ ] 5. Collect data (minimum 2 weeks for weekly patterns)
- [ ] 6. Analyze significance (see [STATISTICAL-ANALYSIS.md](references/STATISTICAL-ANALYSIS.md))
- [ ] 7. Roll out winner or iterate

---

## Statistical Methods (Choose One)

| Method | Best For | Decision Output |
|--------|----------|-----------------|
| **Frequentist** | Fixed sample, strict control | p-value < 0.05 → significant |
| **Bayesian** | Continuous monitoring, intuitive | P(B > A) = 96% → B likely better |
| **Multi-Armed Bandit** | Optimize during test | Auto-shift traffic to winner |
| **Contextual Bandit** | Personalization | Best variant per user segment |

**Quick Bayesian (Beta-Binomial)**:
```python
# A: 50/1000 conversions, B: 72/1000
import scipy.stats as stats
a_samples = stats.beta(51, 951).rvs(100000)  # Beta(1+50, 1+950)
b_samples = stats.beta(73, 929).rvs(100000)  # Beta(1+72, 1+928)
p_b_wins = (b_samples > a_samples).mean()     # → ~0.96 (96%)
```

**Full analysis guide**: [STATISTICAL-ANALYSIS.md](references/STATISTICAL-ANALYSIS.md)

---

## Key Patterns

### Multiple Concurrent Experiments

```tsx
// middleware.ts
const EXPERIMENTS = {
  hero_cta: { weight: 0.5 },
  pricing_layout: { weight: 0.5 },
  signup_flow: { weight: 0.2 },  // 20% on new variant
}

for (const [name, config] of Object.entries(EXPERIMENTS)) {
  if (!request.cookies.get(`exp_${name}`)) {
    const variant = Math.random() < config.weight ? 'B' : 'A'
    response.cookies.set(`exp_${name}`, variant, { maxAge: 2592000, path: '/' })
  }
}

From danzam98/claude-skills-toolkit, MIT.

More from danzam98/claude-skills-toolkit