Alternative Hypothesis

Definition

Alternative Hypothesis is the statement in hypothesis testing that proposes there is a real, measurable difference between the control and treatment variations in an A/B test.

What Alternative Hypothesis means in A/B testing

Denoted as H₁ or Hₐ, the alternative hypothesis is what you're trying to find evidence for in your experiment. It directly opposes the null hypothesis and represents the claim that your variation causes a change in the metric you're measuring. Alternative hypotheses can be one-tailed (directional, predicting improvement or decline) or two-tailed (non-directional, simply predicting a difference). Most A/B tests use one-tailed alternatives because you're specifically testing whether a variation performs better.

Why Alternative Hypothesis matters

Clearly defining your alternative hypothesis before running a test ensures you're measuring the right metrics and sets the foundation for proper statistical analysis. It helps determine your required sample size, informs whether you should use a one-tailed or two-tailed test, and guides the interpretation of results. A well-formulated alternative hypothesis includes the specific metric, direction of change, and ideally the minimum detectable effect you care about.

Example of Alternative Hypothesis

For a button color test, your alternative hypothesis might state: 'Changing the CTA button from blue to red will increase the click-through rate by at least 10% compared to the control.' This stands in contrast to your null hypothesis that there's no difference between the two colors.

How to use Alternative Hypothesis

Use Alternative Hypothesis after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.

Common mistake

A common mistake is treating Alternative Hypothesis as a yes-or-no shortcut while ignoring sample size, test duration, and practical business impact. A statistically interesting result can still be too small, too noisy, or too risky to ship.

Related A/B testing terms

• Alpha
• Bayes Theorem
• Bayesian Statistics
• Beta

FAQ

What does alternative hypothesis mean in A/B testing?

Alternative Hypothesis is the statement in hypothesis testing that proposes there is a real, measurable difference between the control and treatment variations in an A/B test.

Why does alternative hypothesis matter for experiments?

Clearly defining your alternative hypothesis before running a test ensures you're measuring the right metrics and sets the foundation for proper statistical analysis. It helps determine your required sample size, informs whether you should use a one-tailed or two-tailed test, and guides the interpretation of results. A well-formulated alternative hypothesis includes the specific metric, direction of change, and ideally the minimum detectable effect you care about.

How should teams use alternative hypothesis in an experiment?

Use Alternative Hypothesis after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.