Generalization to Aspects of Accuracy Other Than Overconfidence
This concept extends the discussion of overconfidence to the broader notion of calibration—the match between probability judgments and actual frequencies. It defines calibration-in-the-large and the over/underconfidence index, and raises the question of cross-cultural variations in these aspects of judgment accuracy.
THE IDEA
GENERALIZATION TO ASPECTS OF ACCURACY OTHER THAN OVERCONFIDENCE
Overconfidence is a particular variety of “miscalibration” in probability judgments. Calibration refers to the degree of matching that exists between the numbers people report as their probability judgments and the relative frequencies with which the events assigned those numbers actually occur. Suppose that, for each of 50 different days, a weather forecaster says there is a 70% chance of precipitation. If the forecaster is perfectly calibrated, then precipitation will be observed on exactly 35 of those days, a relative frequency of 70%. The match between the overall average judgment and the overall relative frequency is an indication of calibration-in-the-large, a special case of the more general notion of calibration. Suppose the target event is, “My chosen answer is correct,” as in general-knowledge studies. Then the difference between the overall average judgment and the corresponding relative frequency is the “bias” measure we discussed previously, the over/underconfidence index. Virtually everyone intuitively accepts that people should seek good calibration – including minimal overconfidence – in their probability judgments. However, there is much more to probability judgment accuracy than calibration. This motivates our final generality question: Are there reliable cross-cultural variations in other aspects of probability judgment accuracy, and if so, what is their nature?
When to use it
Forecasting weather probabilities to ensure accurate communication of risk.
Assessing confidence in answers on multiple-choice tests and calibrating study strategies.
Evaluating probabilities in investment decisions to avoid over- or underconfidence.
Making predictions in sports analytics where calibrated probability estimates improve betting models.
FROM THEORY TO EVERYDAY LIFE
In practice
Weather Forecasting Calibration
A weather forecaster reports a 70% chance of precipitation on each of 50 days. Perfect calibration would mean precipitation occurs on exactly 35 of those days (70% relative frequency). Calibration-in-the-large measures the match between average probability judgments and actual outcomes.
A MOMENT TO REFLECT
Put it to work.
How can you personally improve your calibration in everyday probability judgments, such as estimating the likelihood of an event?
Think of a situation where your confidence in an answer was high but the actual accuracy was lower. What might explain the mismatch?
How might cultural background influence the way people calibrate their probability estimates?
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)