# A/B test significance calculator > Is your A/B test result real or just chance? Conversion rates, lift, p-value and the confidence interval of an A/B test, with the sample size it needs. Check if variant B really converts better than A: lift, p-value, confidence interval for the difference and how many visitors the test needs. - Page: https://how.box/ab-test/ - Runs entirely in the browser. - License: HowToolbox's own text and data (titles, explanations, FAQs, tool descriptions and schemas, this file) are CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/); AI training and answers are welcome. Attribute "HowToolbox (how.box)" with a link to the page used. Third-party data keeps its own terms; the name and logo are not licensed. Publisher: HowToolbox, contact@how.box. Terms: https://how.box/about/terms/ ## Inputs Answer link: https://how.box/ab-test/?na={na}&ca={ca}&nb={nb}&cb={cb}&c={c}&h={h}&pw={pw} - `na` (whole number 1 to 1e9): Visitors in A. Example: `10000`. - `ca` (whole number 0 to 1e9): Conversions in A. Example: `400`. - `nb` (whole number 1 to 1e9): Visitors in B. Example: `10000`. - `cb` (whole number 0 to 1e9): Conversions in B. Example: `500`. - `c` (number 80 to 99.9): Confidence level in percent. Example: `95`. - `h` (two-sided, greater or less): two-sided, greater or less. Example: `two-sided`. - `pw` (number 50 to 99): Power for the sample-size hint, percent. Example: `80`. Example answer: https://how.box/ab-test/?na=10000&ca=400&nb=10000&cb=500&c=95&h=two-sided&pw=80 A complete link carries the primary answer in the page HTML; a missing or invalid value shows the default page. ## Agent tools - `ab_test_calculate`: A/B test of two conversion rates with the pooled two-proportion z-test (equal to Pearson's χ² without continuity correction). - MCP, one POST to https://how.box/mcp: `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"ab_test_calculate","arguments":{"visitors_a":10000,"conversions_a":400,"visitors_b":10000,"conversions_b":500,"confidence_percent":95,"alternative":"two-sided","power_percent":80}}}` - GET: https://how.box/mcp/run/ab_test_calculate?visitors_a=10000&conversions_a=400&visitors_b=10000&conversions_b=500&confidence_percent=95&alternative=two-sided&power_percent=80 ## How it works An A/B test shows two versions of a page to different visitors and compares how often each one converts. Even two identical pages rarely get exactly the same rate, so the question is whether the gap is bigger than chance alone would usually produce. The test pools both groups into one rate p̂ and compares the difference with its standard error: z = (p_B − p_A) ÷ √(p̂(1 − p̂)(1/n_A + 1/n_B)). The p-value is the chance of a gap at least that large if the two versions really converted equally. The method follows the NIST/SEMATECH e-Handbook; the sample size needed for a given power follows R's power.prop.test. Significance is not the size of the effect: the interval of the difference shows how large or small the true gain could plausibly be. Decide the sample size before the test, run it in full weeks, and don't stop as soon as it turns significant, or chance wins far more often than the confidence level suggests. ## FAQ **What does a p-value of 0.05 mean?** If A and B truly converted at the same rate, a difference at least this large would show up in about 5 % of tests. It is not the chance that B is better. **How many visitors does my A/B test need?** It depends on the rates and the smallest lift worth detecting. Going from 4 % to 5 % needs about 6,745 visitors per variant for 80 % power at 95 % confidence; smaller lifts need many more. **Should I use a one-sided test?** Only if you decided before the test that you care about one direction. A one-sided test reaches significance sooner, but it can't tell you that B made things worse. **The test is not significant. Does that mean A and B are the same?** No. It means the data can't tell them apart yet. With too few visitors, even a real difference often looks like chance. **Can I compare revenue per visitor?** No. This test needs binary outcomes: each visitor either converts or does not. Revenue amounts need a different statistical test. More: [HowToolbox index](https://how.box/llms.txt) · [Agent Skill](https://how.box/skill.md) · [agent terms](https://how.box/about/agents/)