Expert Calibration (Probability Judgments)
The study of how well experts' subjective probabilities match actual outcome frequencies, often revealing overconfidence and poor calibration.
THE IDEA
Expert Calibration
The increasing use of expert judgment during and after World War II helped motivate research into subjective probabilities (Cooke, 1991). Experts are often asked to quantify the likelihood of events such as a stock market collapse, a nuclear plant accident, or a presidential election (Ayton, 1992; Baron, 1998; Hammond, 1996). Calibration concerns how closely those probabilities match observed frequencies across comparable predictions: whether experts are properly “calibrated.” Examining forecasts “in the wild” means studying predictions made in the settings where experts work.
When to use it
Evaluating the accuracy of risk assessments in finance and insurance.
Improving weather forecasting by providing calibrated probability estimates.
Assessing the reliability of expert testimony in legal settings.
Enhancing decision-making in medical diagnosis by quantifying uncertainty.
FROM THEORY TO EVERYDAY LIFE
In practice
Financial Analyst Predictions
A financial analyst repeatedly assigns a 90% probability to a market upturn. Upturns occur in only 60% of those cases, indicating overconfidence in this set of forecasts.
Weather Forecasting
A meteorologist’s rain forecasts are compared with observed rainfall. Among days assigned a 70% chance of rain, rain should occur about 70% of the time for the forecasts to be well calibrated.
A MOMENT TO REFLECT
Put it to work.
In what professional domains are experts most prone to overconfidence in their probabilistic predictions?
What methods can be used to improve the calibration of expert judgments?
How does the heuristics and biases perspective explain poor calibration in expert probability estimates?
References
- Experts in Uncertainty — Cooke, R.M. (1991)
- On the Psychology of Prediction — Ayton, P. (1992)
- Thinking and Deciding — Baron, J. (1998)
- Human Judgment and Social Policy — Hammond, K.R. (1996)
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)