Noise (in Probability Judgments)
Noise refers to variability in probability judgments that is unrelated to the actual occurrence of the target event, often arising from reliance on irrelevant information or from inconsistency. It degrades judgment accuracy separately from bias and has been studied cross-culturally, revealing differences in noise levels among respondents from different countries.
THE IDEA
Noise
Yates (1982) described a "covariance decomposition" of probability judgment accuracy that includes components such as Bias, Base Rate, Slope, and Noise. Noise occurs when variation in probability judgments is unrelated to the event's occurrence or nonoccurrence. Two main sources of noise are:
- Reliance on non-predictive information: Using information that is believed to be related but is actually independent of the target event. For example, a physician using fever to diagnose Disease D when fever and Disease D are uncorrelated.
- Inconsistency or unreliability: The person gives different judgments for cases with identical available information. For instance, diagnosing Patient 15 with a 35% probability for Disease D, and later giving a different probability for Patient 183 who has the same symptoms.
Cross-cultural studies (Yates et al., 1989, 1998) found that Chinese participants' judgments were significantly noisier than American and Japanese participants' judgments, with the latter two groups showing similar noise levels. A follow-up study comparing psychology and engineering students in Taiwan found no difference in overconfidence, suggesting that the cross-cultural effects are not due to differences in academic discipline.
Noise is a distinct factor in judgment accuracy, separate from bias, and varies across individuals and cultures.
When to use it
Evaluating the reliability of employee performance reviews by checking for inconsistency across similar cases.
Auditing medical diagnoses to identify if specific symptoms are being used even when they are not predictive.
Improving forecasting accuracy by reducing reliance on irrelevant information and standardizing judgment processes.
Designing decision-support systems that flag potential noise sources and promote consistent reasoning.
FROM THEORY TO EVERYDAY LIFE
In practice
Physician using fever to diagnose Disease D
A physician believes fever is associated with Disease D and adjusts her probability judgments based on its presence or absence. If fever and Disease D are actually independent, those adjustments are unrelated to the patient's true state, introducing noise and reducing diagnostic accuracy.
Inconsistent diagnosis of identical patients
A physician gives a 35% probability of Disease D for Patient 15, but later gives a different probability for Patient 183 who has identical symptoms. This inconsistency is a source of noise that degrades judgment accuracy.
Cross-cultural noise differences
Yates et al. (1989) found that Chinese participants' general-knowledge judgments were noisier than those of American and Japanese participants, and Yates et al. (1998) replicated this pattern for medical diagnoses, with Chinese respondents showing more unreliability.
A MOMENT TO REFLECT
Put it to work.
In what personal or professional judgments might you be relying on information that is actually unrelated to the outcome? How could you test this?
Can you recall a situation where you gave inconsistent judgments for similar cases? What factors might have caused that noise?
How might noise affect the accuracy of hiring decisions, medical diagnoses, or financial forecasts in your field?
Often confused with
References
- Covariance decomposition of probability judgment accuracy — Yates, J. F. (1982)
- Murphy's decomposition of judgment accuracy — Murphy, A. H. (1973)
- Subjective probability accuracy analysis — Yates, J. F. (1994)
- Noise and judgment accuracy — Yates, J. F. (1998)
- Cross-cultural differences in probability judgment accuracy — Yates, J. F. et al. (1989)
- Noise and inconsistency in medical diagnoses — Yates, J. F. et al. (1998)
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)