Overconfidence in Probabilistic Judgment

THE PSYCHOLOGY OF JUDGMENT UNDER UNCERTAINTY

The premise of laboratory studies of human judgment is that all judgments are governed by a set of core cognitive processes (Tversky & Kahneman, 1981; Kahneman, Slovic, & Tversky, 1982). If those can be understood in experimental settings, then reasonable speculations can be made about human performance in the real world (Fischhoff & Whipple, 1981; Morgan, Henrion, & Morris, 1981). This literature reveals both strengths and weaknesses. When people have the explicit training or there has been the opportunity to receive clear and prompt feedback, people can assess many aspects of uncertain processes. For example, weather forecasters in the United States provide assessments of the probability of precipitation that are probabilistically well calibrated: It rains on about 70% of the occasions on which they forecast a 70% probability of rain (Murphy & Katz, 1983). They have developed this ability through years of hands-on experience, with guidance from computer models, with ample feedback, and within an institution that rewards them for candor (rather than, say, for exuding confidence or avoiding firm commitments). In less favorable circumstances, however, people often lack an intuitive feel for probabilistic processes, relying instead on mental heuristics (deterministic rules of thumb) to guide their judgments. Although often useful, these rules can lead to predictable biases.

The intuitive assessment of uncertainty has proven to be especially problematic (Lichtenstein, Fischhoff, & Phillips, 1982). People seem insufficiently sensitive to how much they know, so that changes in knowledge are accompanied by inappropriate changes in confidence. The most common problem is overconfidence. A common way to assess the precision of an uncertain quantity is by a subjective confidence interval, indicating a range within which the assessor believes the true value has, say, a 98% chance of falling. The probabilistic calibration of a set of such judgments for different quantities may be measured by comparing the assessed probability for the interval with the fraction of times the true value lies within it. Cases in which the true value (once known) turns out to fall outside the assessed confidence interval may be termed “surprises.” The surprise index is the percentage of 98% confidence intervals for which the true value is a surprise. Having significantly more than 2% surprises indicates overconfidence, in the sense of underestimating the range of possibilities. Conversely, too few surprises means underconfidence. Another measure of calibration is the interquartile index, the percent of judgments for which the true value lies between the assessed 25th and 75th percentiles. An interquartile index that is much less than 50% also indicates overconfidence. The overwhelming result in laboratory studies with nonspecialist assessors has been intervals that are too tight, reflecting overconfidence: Typically the surprise index is 20 to 40% instead of the ideal 2%, and the interquartile index is 30 to 40% instead of 50% (Lichtenstein et al., 1982).

The important role of probabilistic judgment is coming to be recognized in several areas involving risk analysis, including medicine, toxicology, and nuclear safety [see, e.g., Raiffa, 1968; Greene & Bourne, 1972; U.S. Nuclear Regulatory Commission (USNRC), 1981; National Academy of Sciences (NAS), 1983]. The few studies of judgments in such real-world contexts outside the psychologist’s laboratory suggest that the laboratory findings of overconfidence may generalize to situations of practical importance (see, e.g., Hynes & Vanmarcke, 1976). However, such evaluations have been rare. An evaluation of the uncertainty estimates for physical constants should therefore be of interest to the study of probabilistic judgment in general, as well as to the practice of physics. . . .