A/B Test Calculator for Creative Tests

How do you use the A/B test calculator?

Enter the successes and the total for each of two ads. The calculator returns both rates, the lift of B over A, the z-score, the two-sided p-value and a significant or not significant verdict.

1. Pick one metric to compare, and use the same one for both ads. For example:
   • Click-through rate: link clicks (successes) out of impressions (total).
   • Hook rate: 3-second video plays out of impressions.
   • Conversion rate: purchases out of link clicks.
2. Enter variant A's successes and total, then variant B's.
3. Read the verdict. The default threshold is a p-value below 0.05, which is 95 percent confidence.
4. Copy the page link to share the result. Only the numbers are stored in it.

Each success must belong to one item in the total. Purchases out of impressions works; purchases out of a different date range's impressions does not.

What method does the calculator use?

It uses a two-sided two-proportion z-test with a pooled standard error. This is the standard test for whether two proportions differ, described in the NIST/SEMATECH e-Handbook of Statistical Methods, section 7.3.3.

Rate A = successes A ÷ total A
Rate B = successes B ÷ total B
Pooled rate = (successes A + successes B) ÷ (total A + total B)
Standard error = √( pooled × (1 − pooled) × (1 ÷ total A + 1 ÷ total B) )
z = (rate B − rate A) ÷ standard error
p = 2 × (1 − Φ(|z|))

Pooling uses both variants to estimate one shared rate, because the test starts from the assumption that the two ads perform the same. Two-sided means the test looks for a difference in either direction, not only "B is better".

Φ is the standard normal cumulative distribution function. The calculator computes it with the rational approximation of the error function in Abramowitz and Stegun, formula 7.1.26, which is accurate to within 1.5 × 10⁻⁷. That is far more precision than any ad test needs.

Worked example

Enter these numbers in the calculator above (example numbers, not from a real test):

The tool shows the same results:

A p-value of 0.0171 means that if the two ads truly converted at the same rate, a gap this large would show up by chance about 1.7 times in 100 tests. That is below the 5 in 100 threshold, so B's higher rate is unlikely to be chance alone.

Significant does not mean large. The test says the difference is probably real; whether a 25 percent lift is worth acting on is a business decision.

What are the limits of this calculator?

The z-test gives a correct answer to a narrow question. Most wrong conclusions from creative tests come from how the test was run, not from the arithmetic.

• Peeking. Checking the result every day and stopping the first time it says significant makes false winners much more likely. Set the sample size or the end date before launch and read the result once.
• Sample size. The z-test uses a normal approximation, which needs enough successes and failures in each variant. With very few conversions, the NIST handbook points to Fisher's exact test instead. In practice: wait for more data.
• More than two variants. Each extra comparison is another chance for a fluke. If you test five hooks against each other, one may look significant by luck. Use a stricter threshold (for example a p-value below 0.01) or test fewer variants at once.
• Uneven audiences. Two ads running side by side in one ad set are not shown to random, separate groups, so they may differ in who saw them, not only in the creative. Meta's A/B testing feature splits the audience evenly and warns that informal tests can lead to overlapping audiences. Use it when the decision matters.
• One metric at a time. A creative can win on CTR and lose on conversion rate. Decide which metric matters before you start.

A creative testing framework covers how to set up tests so these problems are less likely, and the hook rate calculator gives the video metrics you can test with this page.

Related tools and glossary entries

See the CTR and hook rate glossary entries for the two metrics most often compared in creative tests. The CTR calculator and ad metrics calculator give the rates themselves, and our ad hooks guide lists hook types worth testing. All free tools are on the tools page.

If you want more creative variants to test, see how Anglr works for brands.