Power of a Test

Definition

This term refers to the ability of a statistical test to detect a difference when one actually exists. It measures the test’s sensitivity or its capacity to correctly identify true effects. Depending on the context, true effects could mean distinguishing between two different marketing campaigns, product versions, or anything similar.

What Power of a Test means in A/B testing

In an A/B testing workflow, Power of a Test is part of the statistical layer that helps explain whether a result is trustworthy. It is most useful when paired with a clear hypothesis, a primary metric, enough traffic, and a pre-defined decision rule.

Why Power of a Test matters

Power of a Test matters because it helps teams separate real experiment signals from random noise. It should be interpreted alongside sample size, test duration, traffic quality, and the business value of the metric being measured.

Example of Power of a Test

For example, a team testing a new pricing-page headline may see a higher sign-up rate in the variant. Power of a Test helps the team judge whether that lift is strong enough to trust or whether they should keep collecting data before making a decision.

How to use Power of a Test

Use Power of a Test after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.

Common mistake

A common mistake is treating Power of a Test as a yes-or-no shortcut while ignoring sample size, test duration, and practical business impact. A statistically interesting result can still be too small, too noisy, or too risky to ship.

Related A/B testing terms

• Alpha
• Alternative Hypothesis
• Bayes Theorem
• Bayesian Statistics

FAQ

What does power of a test mean in A/B testing?

This term refers to the ability of a statistical test to detect a difference when one actually exists. It measures the test’s sensitivity or its capacity to correctly identify true effects.

Why does power of a test matter for experiments?

Power of a Test matters because it helps teams separate real experiment signals from random noise. It should be interpreted alongside sample size, test duration, traffic quality, and the business value of the metric being measured.

How should teams use power of a test in an experiment?

Use Power of a Test after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.