A/B test significance calculator
Money & financeIs your A/B test result real or just chance?
Conversion rates, lift, p-value and the confidence interval of an A/B test, with the sample size it needs.
What you enter stays on your device while you use this tool. Opening a shared link sends the settings in its address to HowToolbox to prepare the page.
Test settings
Lift of B over A
+25%
Significant at 95% confidence (p < 0.001): a difference this large would be unlikely if A and B converted equally.
More about the test
- Conversion rate A
- 4%
- Conversion rate B
- 5%
- Difference B − A
- +1 percentage points
- 95% interval of the difference
- 0.43 to 1.57 percentage points
- p-value
- < 0.001
- Visitors per variant for 80% power
- 6,745
Significance only says the difference is unlikely to be pure chance. It does not say how big the true lift is, that the change caused it, or that it will last. Decide the sample size before you start and don't stop the test the moment it looks significant.
Sponsored Links
Conversion rates, lift, p-value and the confidence interval of an A/B test, with the sample size it needs.
How to use
- Enter how many visitors saw each variant and how many of them converted.
- Read the lift of B over A and whether the difference is significant; the facts show both rates, the interval of the difference and the p-value.
- Check the visitors each variant needs; open Test settings to change the confidence level, a one-sided test or the power.
Sponsored Links
How it works
An A/B test shows two versions of a page to different visitors and compares how often each one converts. Even two identical pages rarely get exactly the same rate, so the question is whether the gap is bigger than chance alone would usually produce.
The test pools both groups into one rate p̂ and compares the difference with its standard error: z = (p_B − p_A) ÷ √(p̂(1 − p̂)(1/n_A + 1/n_B)). The p-value is the chance of a gap at least that large if the two versions really converted equally. The method follows the NIST/SEMATECH e-Handbook; the sample size needed for a given power follows R's power.prop.test.
Significance is not the size of the effect: the interval of the difference shows how large or small the true gain could plausibly be. Decide the sample size before the test, run it in full weeks, and don't stop as soon as it turns significant, or chance wins far more often than the confidence level suggests.
Worked example
What is the relative lift from 200 to 250 conversions per 5,000 visitors?
- A = 200 ÷ 5,000 = 4%; B = 250 ÷ 5,000 = 5%.
- (B − A) ÷ A = 25%.
Relative lift: 25%. The two-sided test at 95% confidence gives p = 0.01587.
Frequently asked questions
What does a p-value of 0.05 mean?
If A and B truly converted at the same rate, a difference at least this large would show up in about 5 % of tests. It is not the chance that B is better.
How many visitors does my A/B test need?
It depends on the rates and the smallest lift worth detecting. Going from 4 % to 5 % needs about 6,745 visitors per variant for 80 % power at 95 % confidence; smaller lifts need many more.
Should I use a one-sided test?
Only if you decided before the test that you care about one direction. A one-sided test reaches significance sooner, but it can't tell you that B made things worse.
The test is not significant. Does that mean A and B are the same?
No. It means the data can't tell them apart yet. With too few visitors, even a real difference often looks like chance.
Can I compare revenue per visitor?
No. This test needs binary outcomes: each visitor either converts or does not. Revenue amounts need a different statistical test.
Next you might need
- z-score calculatorThe z-score and percentile of a value on a normal distribution, the value at a percentile, or the chance between two values., private, runs on your deviceMath & numbersCalculatorStatistics
- Average calculatorMean, median, mode and range of a list of numbers, with the count, sum, smallest and largest., private, runs on your deviceMath & numbersCalculatorStatistics
- Combinations and permutations calculatorExact counts of combinations and permutations, with or without repetition, and the chance of one particular pick., private, runs on your deviceMath & numbersCalculatorStatistics
- Standard deviation calculatorSample or population standard deviation and variance of a list of numbers, with the mean and the standard error., private, runs on your deviceMath & numbersCalculatorStatistics
- APR calculatorTurn the interest rate and the fees of a loan into one annual percentage rate (APR) by the EU formula., private, runs on your deviceMoney & financeCalculatorLoansInterest ratesCosts
- Break-even calculatorSee the units and sales that cover your fixed and variable costs., private, runs on your deviceMoney & financeCalculatorCosts
More in Money & finance
All 38 utilities in Money & finance- Currency converterConverts between about 30 currencies with the European Central Bank's daily reference rates, and shows the day they were set., private, runs on your device
- Mortgage calculatorSee your monthly payment, total interest and how your loan shrinks over time., private, runs on your device
- Loan calculatorPayment, interest and fees of any loan: annuity or equal principal, monthly or less often., private, runs on your device
- VAT calculatorAdd VAT to a price or take it out, with the standard and reduced VAT rates of your country., private, runs on your device
- Business ID checkerCheck a Finnish business ID's check digit and get the matching VAT number., private, runs on your device
- Car loan calculatorSee the monthly payment and full purchase cost, including your deposit, trade-in, tax and fees., private, runs on your device
Popular now
Sponsored Links
Sponsored Links
Sponsored Links