Overconfidence Bias in Expert Judgments

A Climate Example

A striking feature of the debates over many technical issues is the focus on best estimates, with uncertainties expressed, at best, in terms of verbal quantifiers (e.g., “rarely,” “likely”). Such incomplete treatment of uncertainty might itself reflect incomplete statistical intuitions (Fischhoff & Beyth-Marom, 1983; Funtowicz & Ravetz, 1990; Phillips, 1973). In an effort to clarify disagreements over the climate changes caused by increased atmospheric CO 2, Morgan and Keith (1995) conducted daylong interviews with a dozen leading scientists. By focusing on uncertainties, they tried to introduce Bayesian thinking into a high-stakes domain where it was lacking – eliciting probability distributions over variables like “the change in mean global temperature at 40° N, assuming a doubling of atmospheric CO 2.” Their introduction included a lecture on the psychology of judgment. In order to reduce the risk of overconfidence, the interviewers used cognitively informed procedures, such as starting with extreme fractiles (rather than anchoring on best estimates) and eliciting reasons why proposed values might be right and wrong. In order to improve the match between the task and respondents’ natural ways of thinking, the experts were encouraged to redefine questions to forms that they were more accustomed to answering. (Experts who did so also provided the auxiliary assumptions needed to translate their answers to ones fitting the original questions, so that all respondents’ judgments could be expressed in common terms.) Lest the experts become captive of a single perspective, the interviewers had them triangulate answers to related questions.

These interviews elicited sufficiently precise judgments to be evaluated, once events unfolded. Moreover, they documented the experts’ reasons clearly enough for those events to teach something about the underlying theories. Unfortunately, a long time will pass before the critical events occur, and decisions need to be made now. Unless the beliefs of today’s experts are used well, people of the future will have far worse worries than determining which cognitive processes led to fatefully wrong decisions. Appropriate usage means understanding not only what the experts believe, but also how far to trust them.

In the absence of external events for evaluating experts, there may be internal ones. For example, some experts’ beliefs suggested incoherence, reminiscent of that seen in experimental studies. One form involved inconsistency between holistic judgments (e.g., mean temperature at 40°N) and estimates obtained by combining conditional judgments (e.g., mean temperature at 40°N, with and without the North Atlantic conveyor shutting down) with probabilities for the conditioning events (e.g., the conveyor shutting down, or not). A second form of incoherence involved believing that uncertainty could be greater in the future, despite the intervening research. Such beliefs might reflect the sort of second-order uncertainty that should be incorporated in probability distributions, according to some accounts of Bayesian inference, but might be withheld, according to others (Gärdenfors & Sahlin, 1989; Savage, 1955; Shafer & Tversky, 1985).

For policy purposes, an important result of the study was revealing hitherto undocumented consistency across the different experts’ beliefs. That is, their probability distributions typically overlapped considerably, a fact obscured in discussions of best estimates. Even the distributions of a prominent skeptic (about global warming) typically lay within the ranges established by the other experts. The skeptic’s distributions were, however, much narrower, reflecting much greater confidence (and perhaps overconfidence?). There is reason to believe that the interview process and the credibility of its results benefited from being grounded in psychological research. Time will tell whether the exercise led to better gambles with our environment.