Overconfidence in Judgment: Confirmatory Bias and Ecological Models

Confirmatory Bias and Ecological Models

According to the confirmatory bias model of Koriat, Lichtenstein, and Fischhoff (1980), a general tendency toward overconfidence arises from people’s inclination to recruit reasons from memory that confirm the focal hypothesis. The stronger and more numerous the reasons that are recruited, the greater is the confidence expressed in the selected answer. Because this process inclines the judge to overlook reasons against the selected answer, however, he or she is likely to be overconfident that the selected answer is correct. Consistent with this claim, Koriat et al. reported that asking subjects to generate reasons favoring and opposing both options in a two-alternative forced-choice (2-AFC) task reduced overconfidence relative to a control condition in which no such reasons were generated. Furthermore, asking subjects to generate reasons contradicting their preferred alternative reduced overconfidence relative to a control condition while generation of supporting reasons had no effect.

Experts might be more or less susceptible to confirmatory bias than novices. On the one hand, their extensive experience may help them learn to evaluate evidence in a more impartial manner, and could also lead them to spontaneously generate alternatives to the focal hypothesis. On the other hand, the broader knowledge base of experts might lead them to more readily generate a biased set of reasons favoring the focal hypothesis.

The results of Koriat et al.’s (1980) studies have proven somewhat difficult to replicate (e.g., Fischhoff & MacGregor, 1982). Biased recruitment of evidence may play a more pronounced role when the focal hypothesis is highly self-relevant such that the judge is motivated to confirm (or disconfirm) it. Hoch (1985), for example, reported results consistent with the confirmatory search model in a study of predictions made by business school students about the outcomes of their job searches, all of which were evaluatively positive (e.g., receiving more than three job offers). When compared to the actual job search outcomes, the students’ predictions proved to be substantially overconfident in the sense of overestimating these positive events’ likelihood of occurrence (the pattern of overprediction in Fig. 39.1, which is not included here); furthermore, overprediction was greater for low-baserate events. As predicted by the confirmatory search model, asking students to generate reasons why the target event would not occur substantially reduced this overconfidence, while asking them to generate reasons supporting the target event’s occurrence had no influence. Confirmatory search processes, then, might be apparent only in the assessment of highly self-relevant outcomes, producing optimistic overconfidence (Kahneman & Lovallo, 1993; Kahneman & Tversky, 1995; Taylor & Brown, 1988). Experts may not generally exhibit such an optimistic bias, as the outcomes they assess do not typically concern themselves personally.

Some researchers have argued that the prevalence of overconfidence in two-alternative laboratory tasks is attributable to biased selection of particularly difficult or surprising items, a concern first raised by May (1986). Derived from a Brunswikian framework in which cue-based representations of the environment are constructed from experience (Brunswik, 1943, 1955), ecological models draw attention to the manner in which the events serving as the target of judgment in laboratory tasks are selected or constructed (Björkman, 1994; Gigerenzer et al., 1991; Juslin, 1994). Ecological models are based on an explicit or implicit assumption that people are able to accurately internalize cue-based environmental probabilities. According to the ecological models, people produce these ecological or cue validities when asked for confidence assessments, and should be expected to be well-calibrated as long as test items are selected from the reference class in an unbiased manner. Miscalibration, on this account, is a result of non-representative sampling of test items.

The ecological models predict perfect calibration for tasks in which judgment items are randomly selected from a natural reference class. Consistent with such claims, early comparisons between “selected” and “representative” tasks revealed substantial overconfidence for selected tasks but little or no over-confidence for the representative tasks (Gigerenzer et al., 1991; Juslin, 1994). Critics, however, have noted that such experiments confounded method of item selection with task difficulty (Griffin & Tversky, 1992): The easier, representative tasks would be expected to yield less overconfidence than the more difficult, selected tasks on the basis of difficulty alone. Moreover, a number of studies using representative sampling have nonetheless found overconfidence (e.g., Budescu, Wallsten, & Au, 1997; Griffin & Tversky, 1992; Paese & Sniezek, 1991; Suantak, Bolger, & Ferrell, 1996; Schneider, 1995; Sniezek, Paese & Switzer, 1990). A survey of 25 tasks employing representative sampling (Juslin, Olsson, & Björkman, 1997) reveals a clear pattern of overconfidence for difficult tasks and underconfidence for easy tasks, contrary to the predictions of the ecological models that no such effect should occur under representative sampling.

For full-range tasks, in which people estimate an event’s probability across the full range of possible probabilities, the ecological models imply that there should be no systematic effect of discriminability or target event base rate on calibration; indeed, as long as representative sampling is used, these models predict perfect calibration regardless of level of discriminability or base rate. Both of these variables have been found to influence full-range calibration for general knowledge items (e.g., Lichtenstein et al., 1982; Smith & Ferrell, 1983), contrary to the predictions of the ecological models. One might argue, however, that such laboratory studies fail to satisfy representative design. On-the-job assessments of experts may better reflect the natural reference classes in which their expertise developed. If those caseloads are representative and the models’ learning assumptions hold, ecological models predict good calibration. Representativeness still needs to be examined rather than guaranteed by the label of expert work.

Random support theory (RST) is a model of calibration that describes how support for competing hypotheses relates to probability judgments and outcomes. Although developed in the heuristics and biases tradition, it can represent predictions from several theories through changes in its parameters as base rates and evidence quality vary. The notion of case-based judgment from the heuristics and biases perspective suggests that as the outcome base rate varies across tasks and contexts, the bias parameter β will remain essentially constant (neglect of base rate), and that as the level of evidential weight or discriminability varies, the extremity parameter σ will remain essentially constant (neglect of evidence quality). Case-based judgment implies that support will reflect the evidence related to the particular case at hand, and will not be sensitive to aggregate properties of the set of judgment items. Consequently, the parameters of the RST model that ought to reflect these aggregate properties (β and σ) will remain roughly constant despite changes in base rate or evidence quality.

In contrast, the general confirmatory bias model implies a positive measure of focal bias (β) regardless of the task characteristics, and the special case of optimistic bias is characterized by a positive focal bias for positive events and a negative focal bias for negative events. Ecological models predict that, with representative sampling of items, calibration will be perfect, implying that in ecologically meaningful situations, β will track changes in base rate and σ will track changes in discriminability.