suede-ab-testing

Files (6)

• CARD.md
• SKILL.md
• openai.yaml · agents/
• evals.json · evals/
• sample-size-guide.md · references/
• test-templates.md · references/

SKILL.md (first lines)

# Suede AI A/B Testing Skills

Read [the worked delivery contract](references/worked-contract.md) for a concrete
input, expected artifact, failure response and validation example.

## Compatibility

Works from supplied experiment data and product context; file access for saved plans and a calculator/code runtime for power or statistical calculations. Analytics access is optional when exports are provided.

Use this experimentation playbook to design tests that produce statistically valid, actionable results.

## The Iron Law

```
Predeclare three things before a test launches: sample per variant,
minimum duration, and the decision rule: and read the result only once
all three are satisfied. A result read before then is preliminary.
Never a winner.
```

- **Sample per variant**: the Sample Size table below, or a calculator run on your actual baseline.
- **Minimum duration**: 1 full week (day-of-week variation), 2 business cycles (B2B), through paydays (e-commerce): see the "Minimum Duration Rules" section of [references/sample-size-guide.md](references/sample-size-guide.md).
- **Decision rule**: which metric, at which threshold, decides the call: written down before launch, not after.

Two carve-outs, and only these two:

- A **predeclared sequential or always-valid design** may look early under its own stopping rule (see "Sequential Testing" in the sample-size guide). Declaring it sequential after the peek does not count.
- A **guardrail-triggered stop for harm** is a stop, not a winner call. Kill the variant, report no result.

## Initial Assessment

Check for `.agents/product-marketing.md` (or `.claude/product-marketing.md`, or the legacy `product-marketing-context.md`) and read it if present, baseline conversion rate, traffic volume, and available tooling decide whether a test is even powerable, and they are usually already written down there.

Then work the intake list under Task-Specific Questions below; ask only what the context file did not already answer.

---

## Hypothesis Framework

### Structure

```
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
```

### Example

**Weak**: "Changing the button color might increase clicks."

**Strong**: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."

---

## Test Types

| Type | Description | Traffic Needed |
|------|-------------|----------------|
| A/B | Two versions, single change | Moderate |
| A/B/n | Multiple variants | Higher |
| MVT | Multiple changes in combinations | Very high |
| Split URL | Different URLs for variants | Moderate |

---

## Sample Size

### Quick Reference

| Baseline | 10% Lift | 20% Lift | 50% Lift |
|----------|----------|----------|----------|
| 1% | 170,000/variant | 43,000/variant | 7,800/variant |
| 3% | 54,000/variant | 14,000/variant | 2,600/variant |
| 5% | 32,000/variant | 8,200/variant | 1,500/variant |
| 10% | 15,000/variant | 3,900/variant | 690/variant |

Two-sided, 95% significance, 80% power, relative lift, rounded up.

**Calculators:**
- Two-proportion sample size calculator: https://www.evanmiller.org/ab-testing/sample-size.html
- Optimizely sample size calculator: https://www.optimizely.com/sample-size-calculator/

**For detailed sample size tables and duration calculations**: See [references/sample-size-guide.md](references/sample-size-guide.md)

---

## Metrics Selection

### Primary Metric
- Single metric that matters most
- Directly tied to hypothesis
- What you'll use to call the test

### Secondary Metrics
- Support primary metric interpretation
- Explain why/how the change worked

### Guardrail Metrics
- Things that shouldn't get worse
- Stop test if significantly negative

### Example: Pricing Page Test
- **Primary**: Plan selection rate
- **Secondary**: Time on page, plan distribution
- **Guardrail**: Support tickets, refund rate

---

## Designing Variants

### What to Vary

| Category | Examples |
|----------|----------|
| Headlines/Copy | Message angle, value prop, specificity, tone |
| Visual Design | Layout, color, images, hierarchy |

From JasonColapietro/suede-creator-skills, MIT.

More from JasonColapietro/suede-creator-skills