Bias-Noise Decomposition of Error (MSE)
The mean squared error (MSE) can be decomposed into bias squared plus noise (variance). Bias and noise are independent and equally weighted, so reducing either by the same amount yields the same reduction in overall error. Reducing noise matters even when bias is unknown.
THE IDEA
Figure 7: Two decompositions of MSE
As the mathematical expression and its visual representation both suggest, bias and noise play identical roles in the error equation. They are independent of each other and equally weighted in the determination of overall error. (Note that we will use a similar decomposition into a sum of squares when we analyze the components of noise in later chapters.)
The error equation provides an answer to the practical question that Amy raised: how will reductions in either noise or bias by the same amount affect overall error? The answer is straightforward: bias and noise are interchangeable in the error equation, and the decrease in overall error will be the same, regardless of which of the two is reduced. In figure 4, in which bias and noise happen to be equal (both are 10%), their contributions to overall error are equal.
The error equation also provides unequivocal support for Amy Simkin’s initial impulse to try to reduce noise. Whenever you observe noise, you should work to reduce it! The equation shows that Amy’s boss was wrong when he suggested that GoodSell wait to measure the bias in its forecasts and only then decide what to do. In terms of overall error, noise and bias are independent: the benefit of reducing noise is the same, regardless of the amount of bias.
This notion is highly counterintuitive but crucial. To illustrate it, figure 8 shows the effect of reducing bias and noise by the same amount. To help you appreciate what has been accomplished in both panels, the original distribution of errors (from figure 4) is represented by a broken line.
When to use it
Improving the accuracy of sales or weather forecasts by focusing equally on reducing systematic bias and random noise.
Evaluating the performance of a team of estimators: measuring both their average deviation (bias) and their consistency (noise) to identify the most impactful improvements.
Designing experiments or surveys where reducing measurement error (noise) can be as effective as removing bias in the instrument.
Deciding whether to invest resources in calibrating a model (reducing bias) or increasing sample size (reducing noise) to lower prediction error.
FROM THEORY TO EVERYDAY LIFE
In practice
Amy Simkin's Forecasting Dilemma
Amy Simkin at GoodSell wanted to reduce noise in forecasts, but her boss argued they should first measure bias. The error equation shows that reducing noise is equally beneficial regardless of bias, so Amy was right to prioritize noise reduction.
Equal Bias and Noise
In figure 4 of the text, bias and noise are both 10%, meaning they contribute equally to overall error. A 1% reduction in either yields the same decrease in MSE.
A MOMENT TO REFLECT
Put it to work.
In what real-world situation might you be tempted to ignore noise because you think bias is more important? How does the error equation change your approach?
Think of a process where you can measure overall error but cannot easily separate bias and noise. How can you still effectively reduce error based on this decomposition?
How does the independence of bias and noise affect the strategy for improving a predictive model?
CONTINUE THE CONVERSATION
Source texts
- Noise A Flaw in Human Judgment (Daniel Kahneman, Olivier Sibony etc.)