Find out if your A/B test results are statistically significant or just random noise. Enter your numbers below.
Conversion Rate
3.00%
Conversion Rate
3.70%
With 94.8% confidence, there's not enough data to declare a winner yet. You need at least 95% confidence.
Relative Improvement
+23.3%
B vs. A
P-value
0.0518
Above 0.05 threshold
Confidence Level
94.8%
Below 95% threshold
Test one thing at a time: subject line, sender name, send time, or CTA button. Testing multiple variables at once makes it impossible to know what caused the difference.
Send Variant A to one half and Variant B to the other. Random assignment is critical — if one group is biased in any way, your results won't be trustworthy.
Don't call a winner too early. You need enough conversions in both groups for the result to be meaningful. As a rule of thumb, aim for at least 100 conversions per variant.
Use this calculator to see if the difference is real. A 95% confidence level means there's only a 5% chance the result happened by luck.
Statistical significance tells you whether the difference between two variants is real or just random noise. When you run an A/B test, some variation between the groups is expected — even two identical emails sent to different people will produce slightly different open rates. The question is whether the difference is big enough to rule out chance.
This calculator uses a two-proportion z-test. It pools the conversion rates from both groups, calculates the standard error, and determines a z-score. That z-score maps to a p-value, which represents the probability of seeing results this extreme if there were actually no difference between the variants.
The 95% threshold (p < 0.05) is widely accepted as the standard for declaring significance. It means there's less than a 5% chance that the observed difference is due to random variation. Some teams use 90% for low-risk tests or 99% when the stakes are high — but 95% is the most common starting point.
Keep in mind that statistical significance doesn't tell you whether the improvement is meaningful for your business. A subject line change that lifts open rates by 0.1% might be statistically significant with a huge sample size, but it probably won't move the needle on your revenue. Always pair statistical results with practical judgment.
It depends on your current conversion rate and the size of the improvement you're looking for. As a rough guide, you'll need at least 1,000 emails per variant for subject line tests (where open rates are the metric) and 5,000-10,000 per variant for click or conversion tests. The smaller the expected difference, the more data you need.
Yes, but this calculator handles two-variant tests. For multi-variant tests (A/B/C/D), you'd need to compare each pair separately and adjust for multiple comparisons. In practice, most email marketers stick to two variants to keep things simple and reduce the sample size needed.
A p-value of exactly 0.05 is right on the line. Technically it meets the threshold, but it's a borderline result. If possible, continue collecting data to see if the result strengthens or weakens. A p-value of 0.01 or lower gives much stronger evidence.
Wait until all (or nearly all) emails have been delivered and the recipients have had time to act. For open rates, 24-48 hours is usually enough. For click-through or conversion metrics, you might need 3-7 days depending on your audience. Checking too early leads to unreliable results.
Built-in A/B testing, real-time analytics, and deliverability monitoring — everything you need to send emails that actually get results.
Start Free Trial