Miscalibration of Probabilistic Judgments

CONCLUSION

Our survey of the calibration of probabilistic judgments in applied settings has identified the discriminability of the hypotheses and the base rate likelihood of the outcome as major predictors of miscalibration in experts’ everyday judgments. Consistent with direct support models and the partition model – and with the difficulty effect described by Lichtenstein et al. (1982) – good calibration is found when discriminability is high and base rates are moderate; overprediction is found when base rates are low; underprediction is found when base rates are high; and an extremity bias (what is usually meant by the term over-confidence) is found when discriminability is low and base rates are moderate. Apparently, the prevalence of overprediction and overconfidence in many domains is not a sign that people are intrinsically biased to give high or extreme probabilities, but that important domains are marked by low base rates and by relatively non-diagnostic information (e.g., identification of drug users in transport industries by personal interviews) (Griffin & Varey, 1996).

The observed pattern of miscalibration in expert judgment is consistent with the predictions of the case-based direct support models derived from the literature on judgmental heuristics. Other models fared less well. Contrary to the predictions of ecological models, expert judgments exhibited systematic miscalibration despite representative sampling of judgment items. In the expert data sets we examined, there is little or no indication of a general bias in favor of the focal hypothesis, as implied by the confirmatory bias model. In particular, there was little evidence of optimistic bias in these data sets. Note, however, that most of the judgments were generally not self-relevant. When the issues were extremely self-relevant, such as the patients’ predictions of their own survival, there was considerable optimistic bias shown. Apparently, an optimistic bias can sometimes contribute to but is not a necessary condition for miscalibration of subjective probability. Other stochastic models, like RST, might be able to reproduce the pattern of miscalibration in expert judgments that we described (or any other pattern), with appropriate supplemental assumptions. We find the general framework of support theory in which RST is based, however, to provide a useful and psychologically plausible interpretation of the patterns that we found: Assessments of probability typically reflect a direct translation of the support provided by the evidence for the target hypotheses, with little regard to the reliability of the evidence or the base rate of the outcome.

The identification of subjective probability with psychological support or strength of evidence gives rise to two very different reactions. According to some scholars, personal or subjective probabilities are of little interest because they do not reflect the way that people naturally process uncertainty (which is presumed to be through frequency counts, Gigerenzer, 1994). A second approach is to accept that subjective probabilities are not only natural, but inescapable. A historical review of the use of concepts related to “chance” in more than 500 years of English literature noted that “With one exception, all quotations found . . . are subjective probabilities. They all are expressions of degrees of belief, at least in a poetic sense, that an event will happen” (Bellhouse & Franklin, 1997, p. 80). Consider the following statement, taken from a recent financial column: “Three months ago, I might have put the risk of an Asia-wide credit crunch . . . at less than 10%, now I’d say it is approaching 30% to 40%.” How can this statement about a unique, not to say unprecedented, event be reworded in frequency terms?

Our guess is that there will always be judgments made in terms of subjective probability or odds about unique events, and many such judgments, even by experts, will fail the test of correspondence. Accepting this, these results imply that training in probability and statistics, and in particular, training in the distinction between intuitive support and extensional probability, is essential to improve judgmental calibration in applied fields (see Stewart & Lusk, 1994, for one useful categorization of the skills needed for proper calibration). The attainment of expertise may sometimes imply good discrimination, but it rarely ensures good calibration.

40. Clinical versus Actuarial Judgment

Robyn M. Dawes, David Faust, and Paul E. Meehl

A psychiatric patient displays ambiguous symptoms. Is this a condition best treated by psychotherapy alone or might it also require an antipsychotic medication with occasionally dangerous side effects? An elderly patient complains of memory loss but neurological examination and diagnostic studies are equivocal. The neuropsychologist is asked to administer tests to help rule out progressive brain disease. A medical work-up confirms a patient’s worst fears: He has terminal cancer. He asks the doctor how long he has to put his life in order.

These three brief scenarios illustrate a few of the many situations in which experts are consulted to diagnose conditions or to predict human outcomes. Optimal planning and care often hinge on the consultant’s judgmental accuracy. Whether as physicians, psychiatrists, or psychologists, consultants perform two basic functions in decision-making: they collect and interpret data. Our interest here is in the interpretive function, specifically the relative merits of clinical versus actuarial methods.