The Law of Large Numbers is a statistical principle stating that as sample size increases, the observed average of results will converge toward the true expected value of the population. In A/B testing, it helps teams describe uncertainty, compare variants, and decide whether an observed lift is reliable enough to act on.
The Law of Large Numbers is a statistical principle stating that as sample size increases, the observed average of results will converge toward the true expected value of the population.
In A/B testing, this theorem explains why larger sample sizes produce more reliable and accurate results that better represent actual user behavior. The law guarantees that random fluctuations and outliers have less impact on outcomes as more data is collected. This mathematical principle underpins the requirement for sufficient sample sizes before declaring test winners.
Understanding this law helps practitioners avoid premature test conclusions based on early data that may not represent true performance differences. It explains why tests need adequate traffic and time to reach statistical significance, preventing costly decisions based on misleading early trends. This principle is fundamental to determining appropriate sample sizes and test duration in experimental design.
In the first 100 visitors to an A/B test, Variation B shows a 50% conversion lift, but after 10,000 visitors, the Law of Large Numbers reveals the true effect is only a 5% improvement as early randomness evens out with larger sample size.
Use Law of Large Numbers after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.
A common mistake is treating Law of Large Numbers as a yes-or-no shortcut while ignoring sample size, test duration, and practical business impact. A statistically interesting result can still be too small, too noisy, or too risky to ship.
The Law of Large Numbers is a statistical principle stating that as sample size increases, the observed average of results will converge toward the true expected value of the population. In A/B testing, it helps teams describe uncertainty, compare variants, and decide whether an observed lift is reliable enough to act on.
Understanding this law helps practitioners avoid premature test conclusions based on early data that may not represent true performance differences. It explains why tests need adequate traffic and time to reach statistical significance, preventing costly decisions based on misleading early trends. This principle is fundamental to determining appropriate sample sizes and test duration in experimental design.
Use Law of Large Numbers after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.
This comprehensive checklist covers all critical pages, from homepage to checkout, giving you actionable steps to boost sales and revenue.
Mida.so is a super lightweight A/B testing tool to help you experiment, analyze and implement conversion strategies in minutes.
