System Noise in Forecasting
Noise refers to unwanted variability in judgments that should be identical, measured by standard deviation. The text illustrates how even when bias is unknown, noise can be identified and addressed to improve decision-making systems.
THE IDEA
Figure 3: Distribution of GoodSell’s market share forecasts for one region
We can attach a number to the amount of noise in GoodSell’s forecasting system. We can compute the standard deviation of the forecasts. As its name indicates, the standard deviation represents a typical distance from the mean. In this example, it is 10 percentage points. As is true for every normal distribution, about two-thirds of the forecasts are contained within one standard deviation on either side of the mean—in this example, between a 34% and a 54% market share. Amy now has an estimate of the amount of system noise in the forecasts of market share. (A better noise audit would use several forecasting problems for a more robust estimate, but one is enough for our purpose here.)
As was the case with the executives of the real insurance company of chapter 2, Amy is shocked by the results and wants to take action. The variation prompts Amy to question how consistently the forecasters are implementing their procedures. Amy asks for authority to hire a noise consultant to achieve more uniformity and discipline in her forecasters’ work. Unfortunately, she does not get approval. Her boss’s reply seems sensible enough: how, he asks, could we reduce errors when we don’t know if our forecasts are right or wrong? Surely, he says, if there is a large average error in the forecasts (i.e., a large bias), addressing it should be the priority. Before undertaking anything to improve its forecasts, he concludes, GoodSell must wait and find out if they are correct.
One year after the original noise audit, the outcome that the forecasters were trying to predict is known. Market share in the target region turned out to be 34%. Now we also know each forecaster’s error, which is simply the difference between the forecast and the outcome. The error is 0 for a forecast of 34%, it is 10% for the mean forecast of 44%, and it is −10% for a lowball forecast of 24%.
Figure 4 shows the distribution of errors. It is the same as the distribution of forecasts in figure 3, but the true value (34%) has been subtracted from each forecast. The shape of the distribution has not changed, and the standard deviation (our measure of noise) is still 10%.
When to use it
Evaluating the consistency of hiring decisions across multiple interviewers
Assessing the variability of medical diagnoses among different doctors
Improving the reliability of performance ratings in employee reviews
Auditing financial forecasts to identify unnecessary variation in assumptions
FROM THEORY TO EVERYDAY LIFE
In practice
GoodSell Market Share Forecasts
The GoodSell audit reports a forecast standard deviation of 10 percentage points. Endpoints such as 24% and 64% describe a range, which is a different measure of spread. Once the actual market share of 34% is known, subtracting it from each forecast reveals errors without changing the standard deviation.
Insurance Company Executives
The authors compare Amy’s reaction with that of insurance-company executives surprised by variation in their judgments. The insurance case is referenced here without its full details.
A MOMENT TO REFLECT
Put it to work.
In what areas of your professional life might noise be present but overlooked because you focus only on bias?
Why do people often assume that reducing bias is more important than reducing noise, even when noise is measurable?
How could you conduct a noise audit in your own decision-making processes to uncover unwanted variability?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- Noise A Flaw in Human Judgment (Daniel Kahneman, Olivier Sibony etc.)