Sample Size

Definition

Sample size refers to the number of individual data points or subjects included in a study or experiment. In the context of A/B testing or marketing, the sample size is the total number of people or interactions (like email opens, webpage visits, or ad viewers) you measure to gather data for your test or analysis. A larger sample size can lead to more accurate results because it offers a more representative snapshot of your overall audience or market.

What Sample Size means in A/B testing

In an A/B testing workflow, Sample Size is part of the statistical layer that helps explain whether a result is trustworthy. It is most useful when paired with a clear hypothesis, a primary metric, enough traffic, and a pre-defined decision rule.

Why Sample Size matters

Sample Size matters because it helps teams separate real experiment signals from random noise. It should be interpreted alongside sample size, test duration, traffic quality, and the business value of the metric being measured.

Example of Sample Size

For example, a team testing a new pricing-page headline may see a higher sign-up rate in the variant. Sample Size helps the team judge whether that lift is strong enough to trust or whether they should keep collecting data before making a decision.

How to use Sample Size

Use Sample Size after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.

Common mistake

A common mistake is treating Sample Size as a yes-or-no shortcut while ignoring sample size, test duration, and practical business impact. A statistically interesting result can still be too small, too noisy, or too risky to ship.

Related A/B testing terms

• Alpha
• Alternative Hypothesis
• Bayes Theorem
• Bayesian Statistics

FAQ

What does sample size mean in A/B testing?

Sample size refers to the number of individual data points or subjects included in a study or experiment. In the context of A/B testing or marketing, the sample size is the total number of people or interactions (like email opens, webpage visits, or ad viewers) you measure to gather data for your test or analysis.

Why does sample size matter for experiments?

Sample Size matters because it helps teams separate real experiment signals from random noise. It should be interpreted alongside sample size, test duration, traffic quality, and the business value of the metric being measured.

How should teams use sample size in an experiment?

Use Sample Size after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.