Difficulty Effect in Confidence Calibration

The Effect of Difficulty

One account suggests that people assess their confidence in one of two competing hypotheses on the basis of their balance of arguments for and against this hypothesis, with insufficient regard for the quality of the data. This mode of judgment gives rise to overconfidence when people form a strong impression on the basis of limited knowledge and underconfidence when people form a moderate impression on the basis of extensive data.

The application of this analysis to general-knowledge questions is complicated by the fact that strength and weight cannot be experimentally controlled as in Studies 1–3. However, in an analogy to a chance setup, let us suppose that the balance of arguments for a given knowledge problem can be represented by the proportion of red and white balls in a sample. The difficulty of the problem can be represented by the discriminability of the two hypotheses; that is, the difference between the probability of obtaining a red ball under each of the two competing hypotheses. Naturally, the greater the difference, the easier the task; that is, the higher the posterior probability of the more likely hypothesis on the basis of any given sample. Suppose confidence is given by the balance of arguments; that is, the proportion of red balls in the sample. What is the pattern of results predicted by this model?

Figure 13.6 displays the predicted results (for a sample of size 10) for three pairs of hypotheses that define three levels of task difficulty: an “easy” task where the probability of getting red balls under the competing hypotheses are .50 and .40, respectively; a “difficult” task, in which the probabilities are .50 and .45; and an “impossible” task, in which the probability of drawing a red ball is .5 under both hypotheses. We have chosen nonsymmetric hypotheses for our example to allow for an initial bias that is often observed in calibration data.

Figure 13.6. Predicted calibration for item difficulty.

It is instructive to compare the predictions of this model to the results of Lichtenstein and Fischhoff (1977) who investigated the effect of task difficulty (Fig. 13.7). Their “easy” items (accuracy = 85%) produced underconfidence through much of the confidence range, their “difficult” items (accuracy = 61%) produced overconfidence through most of the confidence range, and their “impossible” task (discriminating European from American handwriting, accuracy = 51%) showed dramatic overconfidence throughout the entire range.

Figure 13.7. Calibration plots for item difficulty.

A comparison of Figs. 13.6 and 13.7 reveals that our simple chance model reproduces the pattern of results observed by Lichtenstein and Fischhoff (1977): slight underconfidence for very easy items, consistent overconfidence for difficult items, and dramatic overconfidence for “impossible” items. This pattern follows from the assumption that judged confidence is controlled by the balance of arguments for the competing hypotheses. The present account, therefore, can explain the observed relation between task difficulty and overconfidence (see Ferrell & McGoey, 1980).

The difficulty effect is one of the most consistent findings in the calibration literature (Lichtenstein & Fischhoff, 1977; Lichtenstein, Fischhoff, & Phillips, 1982; Yates, 1990). It is observed not only in general-knowledge questions, but also in clinical diagnoses (Oskamp, 1962), predictions of future events (contrast Fischhoff & MacGregor, 1982, versus Wright & Wisudha, 1982), and letter identification (Keren, 1988). Moreover, the difficulty effect may contribute to other findings that have been interpreted in different ways. For example, Keren (1987) showed that world-class bridge players were well calibrated, whereas amateur players were overconfident. Keren interpreted this finding as an optimism bias on the part of the amateur players. In addition, however, the professionals were significantly more accurate than the amateurs in predicting the outcome of bridge hands and the difference in difficulty could have contributed to the difference in overconfidence.

The difficulty effect can also explain the main finding of a study by Gigerenzer, Hoffrage, and Kleinbolting (1991). In this study, subjects in one group were presented with pairs of cities and asked to choose the city with the larger population and indicate their confidence in each answer. The items were randomly selected from a list of all large West German cities. Subjects in a second group were presented with general-knowledge questions (e.g., “Was the zipper invented before or after 1920?”), instructed to choose the correct answer, and to assess their confidence in that answer. Judgments about the population of cities were fairly well calibrated, but responses to the general-knowledge questions exhibited overconfidence. However, the two tasks were not equally difficult: average accuracy was 72% for the city judgments and only 53% for the general-knowledge questions. Hence, the presence of overconfidence in the latter but not in the former could be entirely due to the difficulty effect, documented by Lichtenstein and Fischhoff (1977). Indeed, when Gigerenzer et al. (1991) selected a set of city questions that were matched in difficulty to the general-knowledge questions, the two domains yielded the same degree of overconfidence. The authors did not acknowledge the fact that their study confounded item generation (representative versus selective) with task difficulty (easy versus hard). Instead, they interpreted their data as confirmation for their theory that overconfidence in individual judgments is a consequence of item selection and that it disappears when items are randomly sampled from some natural environment.