Quick answer: Email A/B testing means sending two versions of an email that differ by exactly one variable to comparable random segments, then keeping the winner. The two rules that decide whether it works: test one thing at a time — “if there is more than one difference between your control and variable emails, you will not know what change moved the needle” — and use a big enough sample, because “with only a few hundred contacts, the results will not reach statistical significance” (Litmus). This guide covers what to test, how to size a test, and the mistakes that produce fake winners. Written with AI assistance and reviewed against primary sources.
Most email A/B tests are a waste of time — not because testing does not work, but because the test was too small, too short, or changed three things at once. Here is how to run tests you can actually trust.
What is email A/B testing?
A/B testing (or split testing) sends version A to one random slice of your list and version B to another, changing a single element between them. Whichever version wins on your chosen metric becomes the version you send to everyone else.
The discipline is in the constraint. Litmus is blunt about it: “Unless you are doing multivariate testing, you are going to want to keep your A/B tests limited to one change per test. Having more than one difference between versions A and B makes it difficult to clearly determine which element led to the difference in performance” (Litmus).
What should you test?
Subject lines — the highest-traffic test, since it gates everything downstream. Length matters: per the Litmus 2024 State of Email report, 41–50 characters is the sweet spot for mobile display. For angles to test, see email subject line formulas.
Sender name — often more predictive of opens than the subject line itself. A person name vs. a brand name is a clean single-variable test.
Call to action — button copy, placement, or a button vs. a text link. Test on clicks, not opens.
Send time — a real variable, but one that needs a large list to resolve.
Content format — HTML vs. plain text, long vs. short, image-heavy vs. text-first.
Offer — free shipping vs. a percentage discount is a genuine business test, not a cosmetic one.
How big does an A/B test need to be?
This is where most tests fail. Litmus recommends aiming for 10,000 people or more on a test if you can, warning that a few hundred contacts will not reach statistical significance (Litmus). HubSpot guidance is a floor of roughly 1,000 subscribers per variant for statistically significant results.
The honest math: required sample depends on your baseline rate and the size of the lift you want to detect. Detecting a 20% relative improvement on a 40% open rate takes roughly 592 subscribers per variation — but detecting a small lift on a low baseline can take tens of thousands. Small lifts need big lists.
Aim for 95% statistical confidence before you act on a result, and give the test time: an A/B test “takes at least 48 hours to run its course, and even then, you may want to wait longer to achieve statistical significance” (Litmus).
How to run an A/B test, step by step
1. Start with a hypothesis, not a whim. A first-name subject line will lift opens because our audience knows us personally — that is testable. Let us try something different is not.
2. Change exactly one variable. Everything else — send time, segment, offer — stays identical.
3. Split randomly and evenly. Your two groups must be comparable. Most platforms randomize for you; do not hand-pick segments.
4. Pick the metric before you send. Subject-line tests are judged on opens; CTA tests on clicks; offer tests on conversions or revenue. Choosing the metric after you see results is how people fool themselves.
5. Wait for the data. At least 48 hours, and until you hit your confidence threshold.
6. Act, then re-test. Roll the winner out, document it, and test the next variable. One test is a data point; a testing habit is a competitive advantage.
Mistakes that produce fake winners
Calling a race that is too close. You cannot assume one version won because it is slightly ahead — 22% opens vs. 21% is noise, not a result, unless your sample is large enough to say otherwise.
Testing multiple variables at once. You will get a winner and learn nothing about why.
Stopping early. Peeking and stopping the moment a variant leads is one of the most common ways to manufacture a false positive.
Testing trivia. Button colour rarely moves revenue. Test offers, audiences, and value propositions — the things with real upside.
Ignoring the segment. A test on a poorly targeted list measures your targeting, not your creative. Fix segmentation first — Mailchimp data shows segmented campaigns get 100.95% higher clicks than non-segmented ones, a far bigger lift than most A/B tests will ever find.
Frequently asked questions
What is A/B testing in email marketing? Sending two versions differing by one variable to comparable random groups, then keeping the better performer.
How many subscribers do I need to A/B test? Litmus suggests aiming for 10,000+ on a test; the HubSpot floor is about 1,000 per variant. Below a few hundred, results are not reliable (Litmus).
How long should an A/B test run? At least 48 hours, and longer if you have not reached statistical significance (Litmus).
Can I test more than one thing at once? Only with multivariate testing, which needs a much larger list. Otherwise, one variable per test (Litmus).
What should I test first? Subject line or sender name — they gate every downstream metric and are the easiest clean single-variable tests.
Getting started
A/B testing rewards discipline over cleverness: one variable, a big enough sample, a metric chosen in advance, and the patience to wait 48 hours. If your list is too small to reach significance, do not fake it — spend that effort on segmentation and offers, where the lifts are larger and do not require statistics to see.
For the bigger picture, read the complete email marketing guide, sharpen your targeting with email segmentation, get test ideas from email subject line formulas, or compare tools in Best Email Marketing Software in 2026.
— Shivam