Loss Function

Definition

A loss function quantifies the cost or negative consequence of making a wrong decision in A/B testing, typically measuring the expected loss in revenue, conversions, or other key metrics that would result from choosing an inferior variation.

What Loss Function means in A/B testing

In Bayesian A/B testing, loss functions formalize the business impact of decision errors by calculating the opportunity cost of selecting the worse variation. Different loss functions can reflect different business priorities, such as maximizing conversions, minimizing downside risk, or optimizing revenue. The expected loss for each decision (implementing A or B) is calculated across the entire posterior distribution.

Why Loss Function matters

Loss functions connect statistical analysis directly to business outcomes, enabling data-driven decisions based on actual costs rather than arbitrary significance thresholds. They help determine when to stop a test by quantifying whether the potential gain from continuing outweighs the certain cost of delayed implementation. Using loss functions allows teams to make economically rational decisions that balance statistical uncertainty against business impact.

Example of Loss Function

In testing two pricing strategies, your loss function calculates that if you incorrectly choose the worse option, you'd lose an expected $15,000 per month in revenue. When the expected loss of choosing either variation drops below your $2,000 threshold, you have sufficient confidence to make a decision.

How to use Loss Function

Use Loss Function after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.

Common mistake

A common mistake is treating Loss Function as a yes-or-no shortcut while ignoring sample size, test duration, and practical business impact. A statistically interesting result can still be too small, too noisy, or too risky to ship.

Related A/B testing terms

• Alpha
• Alternative Hypothesis
• Bayes Theorem
• Bayesian Statistics

FAQ

What does loss function mean in A/B testing?

A loss function quantifies the cost or negative consequence of making a wrong decision in A/B testing, typically measuring the expected loss in revenue, conversions, or other key metrics that would result from choosing an inferior variation.

Why does loss function matter for experiments?

Loss functions connect statistical analysis directly to business outcomes, enabling data-driven decisions based on actual costs rather than arbitrary significance thresholds. They help determine when to stop a test by quantifying whether the potential gain from continuing outweighs the certain cost of delayed implementation. Using loss functions allows teams to make economically rational decisions that balance statistical uncertainty against business impact.

How should teams use loss function in an experiment?

Use Loss Function after you have chosen a primary metric and collected enough traffic for a reliable read. Avoid checking it in isolation; compare it with effect size, confidence, practical impact, and whether the test ran long enough to cover normal traffic patterns.