Stochastic Models for Probability Calibration
The authors favor Reference Signal Theory (RST) over two other stochastic accounts of probability calibration, arguing that it requires fewer assumptions and has parameters that are easier to interpret.
THE IDEA
OTHER STOCHASTIC MODELS
Two alternatives to Reference Signal Theory (RST) use stochastic components to reproduce probability-calibration curves. With additional assumptions, both can reproduce the expert-judgment patterns reviewed by the authors. The authors favor RST because they find it more parsimonious and its parameters easier to interpret psychologically. The alternatives are neither named nor explained in this excerpt.
When to use it
Calibrating probability estimates in financial risk assessment
Assessing the calibration of clinicians’ confidence in diagnoses.
Evaluating and debiasing expert forecasts in intelligence analysis
Comparing different models of probability judgment in psychological research
FROM THEORY TO EVERYDAY LIFE
In practice
Weather Forecasting
A meteorologist predicts a 70% chance of rain. Over many such predictions, the fraction of days with rain when 70% was forecast is compared to 70%; deviations indicate miscalibration. Stochastic models aim to capture systematic deviations like overconfidence or underconfidence.
Medical Diagnosis
A doctor estimates a 90% probability that a patient has a certain disease. If across all 90% confidence diagnoses the actual disease rate is only 80%, the doctor is overconfident. Stochastic models reproduce such patterns with assumptions about judgment noise.
A MOMENT TO REFLECT
Put it to work.
How could adding stochastic components help explain why experts are often overconfident or underconfident?
Why is parsimony important when choosing between calibration models?
What psychological mechanisms might underlie the parameter values in RST compared to other stochastic models?
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)