ab-testing-statistics-ai-ml-stub

Files (1)

• SKILL.md

SKILL.md (first lines)

## Overview

Run statistically valid experiments: from pre-experiment power analysis through post-experiment significance testing, variance reduction, and decision reporting. Covers both frequentist and Bayesian approaches.

## When to Use

- Designing a new A/B or multivariate experiment
- Calculating required sample size before launching
- Analyzing results for statistical significance
- Reducing variance to detect smaller effects faster
- Choosing between frequentist p-values and Bayesian posteriors
- Detecting sample ratio mismatch (SRM) or data quality issues

## Sample Size Calculation

Always calculate before launching. Use two-proportion z-test for binary metrics, t-test for continuous.

```python
from scipy import stats
import numpy as np

def sample_size_two_proportion(p_control, mde, alpha=0.05, power=0.80):
    """
    p_control: baseline conversion rate
    mde: minimum detectable effect (relative), e.g. 0.05 for 5%
    Returns: n per variant
    """
    p_treatment = p_control * (1 + mde)
    p_pooled = (p_control + p_treatment) / 2
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(power)
    n = (z_alpha + z_beta)**2 * (p_control*(1-p_control) + p_treatment*(1-p_treatment)) / (p_treatment - p_control)**2
    return int(np.ceil(n))

# Example: 5% baseline, detect 10% relative lift, α=0.05, 80% power
n = sample_size_two_proportion(p_control=0.05, mde=0.10)
# → ~3,744 per variant
```

## Significance Testing

**Binary metrics (conversion rate):**
```python
from statsmodels.stats.proportion import proportions_ztest

def test_proportions(conversions_c, n_c, conversions_t, n_t):
    counts = np.array([conversions_t, conversions_c])
    nobs = np.array([n_t, n_c])
    stat, p_value = proportions_ztest(counts, nobs)
    return stat, p_value
```

**Continuous metrics (revenue, time on page):**
```python
from scipy.stats import ttest_ind

def test_continuous(control_values, treatment_values):
    stat, p_value = ttest_ind(control_values, treatment_values, equal_var=False)  # Welch's
    effect_size = (np.mean(treatment_values) - np.mean(control_values)) / np.std(control_values)
    return stat, p_value, effect_size
```

## CUPED — Variance Reduction

CUPED (Controlled-experiment Using Pre-Experiment Data) reduces variance using pre-experiment covariates, enabling smaller sample sizes or faster detection.

```python
def cuped_adjustment(Y, X):
    """
    Y: outcome metric (post-experiment)
    X: pre-experiment covariate (same metric, pre-period)
    Returns: variance-reduced Y_cuped
    """
    theta = np.cov(Y, X)[0, 1] / np.var(X)
    Y_cuped = Y - theta * (X - np.mean(X))
    variance_reduction = 1 - np.var(Y_cuped) / np.var(Y)
    print(f"Variance reduced by {variance_reduction:.1%}")
    return Y_cuped

# Then run standard t-test on Y_cuped values per variant
```

Typical variance reduction: 20-50%. Requires: pre-experiment data for the same users, covariate correlated with outcome (r > 0.3 helps).

## Multi-Armed Bandits (Thompson Sampling)

Use when exploration-exploitation trade-off matters more than clean causal inference. Not appropriate when you need an unbiased p-value.

```python
import numpy as np

class ThompsonSamplingBeta:
    """For binary reward (click/no-click, convert/no-convert)."""
    def __init__(self, n_arms):
        self.alpha = np.ones(n_arms)  # successes + 1
        self.beta = np.ones(n_arms)   # failures + 1

    def select_arm(self):
        samples = np.random.beta(self.alpha, self.beta)
        return np.argmax(samples)

    def update(self, arm, reward):
        self.alpha[arm] += reward
        self.beta[arm] += (1 - reward)

    def winning_probability(self, n_samples=10000):
        """Monte Carlo estimate of P(arm_i is best)."""
        samples = np.random.beta(self.alpha[:, None], self.beta[:, None], (len(self.alpha), n_samples))
        wins = np.argmax(samples, axis=0)
        return np.bincount(wins, minlength=len(self.alpha)) / n_samples
```

## Sample Ratio Mismatch (SRM) Detection

SRM means traffic split is not what you configured — invalidates results. Always check before analyzing.

```python
def detect_srm(observed_counts, expected_ratios, alpha=0.01):
    """

From UltronCore/claude-skill-vault, MIT.

More from UltronCore/claude-skill-vault