Statistical Power

Definition

Statistical Power is the probability that a test will correctly reject a false null hypothesis. In other words, it's the likelihood that if there actually is a difference (in the case of A/B testing, a difference between the two versions being tested), the test will detect it. A test with a high statistical power is more reliable and less likely to produce false negative results.

What Statistical Power means in A/B testing

In an A/B testing workflow, Statistical Power is part of the statistical layer that helps explain whether a result is trustworthy. It is most useful when paired with a clear hypothesis, a primary metric, enough traffic, and a pre-defined decision rule.

Why Statistical Power matters

Statistical Power matters because it helps teams separate real experiment signals from random noise. It should be interpreted alongside sample size, test duration, traffic quality, and the business value of the metric being measured.

Example of Statistical Power

For example, a team testing a new pricing-page headline may see a higher sign-up rate in the variant. Statistical Power helps the team judge whether that lift is strong enough to trust or whether they should keep collecting data before making a decision.

How to use Statistical Power

Use Statistical Power after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.

Common mistake

A common mistake is treating Statistical Power as a yes-or-no shortcut while ignoring sample size, test duration, and practical business impact. A statistically interesting result can still be too small, too noisy, or too risky to ship.

Related A/B testing terms

• Alpha
• Alternative Hypothesis
• Bayes Theorem
• Bayesian Statistics

FAQ

What does statistical power mean in A/B testing?

Statistical Power is the probability that a test will correctly reject a false null hypothesis. In other words, it's the likelihood that if there actually is a difference (in the case of A/B testing, a difference between the two versions being tested), the test will detect it.

Why does statistical power matter for experiments?

Statistical Power matters because it helps teams separate real experiment signals from random noise. It should be interpreted alongside sample size, test duration, traffic quality, and the business value of the metric being measured.

How should teams use statistical power in an experiment?

Use Statistical Power after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.