Null Hypothesis

Definition

A null hypothesis is a statistical concept that assumes there is no significant difference or relation between certain aspects of a study or experiment. In other words, it's the hypothesis that your test is aiming to disprove. For example, in an A/B test, the null hypothesis might be that there's no difference in conversion rates between version A and version B of a webpage.

What Null Hypothesis means in A/B testing

In an A/B testing workflow, Null Hypothesis is part of the statistical layer that helps explain whether a result is trustworthy. It is most useful when paired with a clear hypothesis, a primary metric, enough traffic, and a pre-defined decision rule.

Why Null Hypothesis matters

Null Hypothesis matters because it helps teams separate real experiment signals from random noise. It should be interpreted alongside sample size, test duration, traffic quality, and the business value of the metric being measured.

Example of Null Hypothesis

For example, a team testing a new pricing-page headline may see a higher sign-up rate in the variant. Null Hypothesis helps the team judge whether that lift is strong enough to trust or whether they should keep collecting data before making a decision.

How to use Null Hypothesis

Use Null Hypothesis after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.

Common mistake

A common mistake is treating Null Hypothesis as a yes-or-no shortcut while ignoring sample size, test duration, and practical business impact. A statistically interesting result can still be too small, too noisy, or too risky to ship.

Related A/B testing terms

• Alpha
• Alternative Hypothesis
• Bayes Theorem
• Bayesian Statistics

FAQ

What does null hypothesis mean in A/B testing?

A null hypothesis is a statistical concept that assumes there is no significant difference or relation between certain aspects of a study or experiment. In other words, it's the hypothesis that your test is aiming to disprove.

Why does null hypothesis matter for experiments?

Null Hypothesis matters because it helps teams separate real experiment signals from random noise. It should be interpreted alongside sample size, test duration, traffic quality, and the business value of the metric being measured.

How should teams use null hypothesis in an experiment?

Use Null Hypothesis after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.