A/B Testing Email Campaigns: What to Test First

Most “A/B tests” in ecommerce email are two subject lines, 800 recipients, and a story in Slack. That is not a test. That is a coin flip with branding.

A/B testing email campaigns is useful after the system exists: segments, flows, a calendar that is not all 20% off. Before that, you are optimizing a leak.

This is the order we use so tests compound instead of generating folklore.

Decide the job of the email before you split it

A welcome email is trying to create a first order. A cart email is trying to recover a checkout. A Tuesday campaign might be trying to sell a drop or tell a story. If you cannot name the job in one sentence, you cannot name the metric.

Opens are a weak primary after Apple privacy. Use them as a sanity check. Full rant: email open rates guide.

Test one thing, in this order

If everything is variable, you will “learn” nothing. Sequence:

1. Audience. VIP vs everyone. Buyers vs non-buyers. This is usually a bigger lift than a pun in the subject. It is also a strategy test, not a copy test.
2. Offer / merch. Product vs product, code vs no code, early access vs public. This is where money lives.
3. Hook (subject + preheader). Specific vs vague. After 1 and 2 are not stupid.
4. Body. Hero, CTA wording, one product vs three. After the hook is not stupid.
5. Send time. Last, and only with volume. Tuesday 10 a.m. winning once is not a company policy.

Bad:  subject + hero + 15% vs 20% + send time, 1,200 people
Good: same email, buyers only, 15% vs no code, wait for orders

Klaviyo will let you test subjects on 10% of a list and auto-send the winner on opens. That feature is convenient and often wrong. If you use it, winner by clicks or placed orders, not opens, and only on lists big enough that 10% is not 90 people.

Volume, time, and the lie of 95% confidence

You need enough orders, not enough opinions.

Rules of thumb we actually use:

• Do not call a campaign test unless each variant can reasonably get tens of orders, not tens of clicks, if revenue is the job.
• Flows: let a test run through a full cycle (a week of welcome traffic, not Tuesday afternoon).
• Do not stack tests on the same audience the same week.
• Stop peeking at hour three.

If the list is small, run fewer tests and longer holdouts. A 10% holdout on a campaign teaches you more than six subject tests a month.

What we see go wrong

Testing discounts against discounts. 10% vs 15% “winner” becomes 15% forever. You trained the list. That is not optimization. That is a hole in margin. Test code vs no code, or code vs a gift-with-purchase, not an endless ladder.

Testing on unengaged people. They do not represent buyers. Exclude them. See deliverability.

Changing the template mid-test. You invalidated the week.

Declaring a brand voice winner from one send. Voice is a system. One subject is a data point.

No QA. Version B “won” because version A’s button was broken on iOS. Mobile email design first.

A four-week test calendar

Week 1. Pick one flow (usually cart email 1 or welcome email 1). One variable. Write the hypothesis in a sentence: “A specific product in the subject will beat a generic ‘you left something’ on placed orders.”

Week 2. Let it run. Do not edit.

Week 3. Read orders, not vibes. Ship the winner or revert. Document it in one paragraph so next month’s you does not retest it.

Week 4. Campaign test: merch or offer, not punctuation. Same segment as last month’s control if you can.

That is enough science for a brand that also has to ship product.

If you want tests ranked by revenue leak — not a culture of random splits — talk to SuperMailo.