Scientific evaluation of your marketing experiments. Determine winners with statistical confidence using Frequentist and Bayesian analysis.
Start by defining the core elements of your experiment. A clear name and focused goal ensure your results are actionable.
Describe what each variant represents. This helps you interpret results and share findings with your team.
💡 Tip: Only test one variable at a time. If you change multiple elements (image, headline, CTA), you won't know what caused the difference.
Enter the visitors and conversions each variant received during your test.
Analysis of your experiment with statistical confidence.
Please wait.
—| Z-Score | — |
| P-Value (two-tailed) | — |
| Confidence Level | — |
| Statistical Significance | — |
| Effect Size (Relative Lift) | — |
| Probability B > A | — |
| Expected Loss if B Chosen | — |
| Risk Assessment | — |
| Practical Recommendation | — |
💡 Bayesian analysis provides a probability that Variant B is better than Variant A. This is often more intuitive for marketers than a p-value.
—
Copy your test results for reporting or save them for future reference.
If your test results are inconclusive or you need help deciding what to do next, reach out. I can help you structure your next experiment.
Discuss Your Testing Setup →A/B testing (also called split testing) compares two versions of a page, ad, email, or creative to see which performs better. You show Version A to half your audience and Version B to the other half, then measure which one drives more conversions.
The problem is that most marketers see a 5% improvement in Version B and immediately declare it a winner. But a 5% improvement over a small sample could just be random variation.
This tool tells you the truth: Whether your observed difference is statistically significant — or whether you need more data before making a decision.
This tool provides both methods so you can make decisions with confidence.
Uses p-values and confidence levels. Asks: "If there is actually no difference between A and B, what is the probability of seeing this result by chance?"
Provides the probability that Variant B is actually better than Variant A, based on observed data.
💡 Best practice: Use both. If frequentist says "significant" AND Bayesian says "high probability B > A," you can be very confident in rolling out Variant B.
The required sample size depends on:
| Baseline CR | Target Lift | Visitors Per Variant |
|---|---|---|
| 2% | +20% | ~35,000 |
| 2% | +50% | ~6,500 |
| 5% | +20% | ~10,000 |
| 5% | +50% | ~2,000 |
| 10% | +20% | ~4,500 |
| 10% | +50% | ~900 |
💡 Rule of thumb: If your baseline CR is under 3%, do not run tests for less than 2 weeks. You need enough data to see past normal week-to-week variance.
❌ Peeking at the results too early: Stopping the test as soon as B leads guarantees false positives about 30% of the time. Decide the sample size upfront, then wait.
❌ Ignoring sample size: 100 visitors and 5 conversions don't prove anything. You need enough data for statistical confidence.
❌ Testing too many things at once: Change one thing per test. If you change 5 things, you don't know what worked.
❌ Not testing against a control: Testing B vs C without A doesn't tell you if either is better than what you have. Always include your current version.
❌ Trusting a 1% lift: A 1% lift is statistically meaningless in most cases. Focus on tests that could realistically produce 10%+ gains.
A/B testing without statistical discipline creates the illusion of progress. If you're running tests but not sure whether the results actually mean anything — let's look at it together.