Illusion of Validity

Frequency versus Confidence

The relation between people’s confidence in the validity of their individual answers and their estimates of their overall hit rate reveals a consistent discrepancy. A sportscaster, for example, can be asked to assess confidence in the prediction of each game as well as the number of games he expects to predict correctly. According to the present account, these judgments are not expected to coincide because they are based on different evidence. A judgment of confidence in a particular case depends primarily on the balance of arguments for and against a specific hypothesis—for example, the relative strength of two opposing teams. In contrast, the estimated frequency of correct prediction is likely to be based on a general evaluation of the difficulty of the task, the knowledge of the judge, or past experience with similar problems. Thus, the overconfidence observed in average judgments of confidence need not apply to global judgments of expected accuracy.

One perspective (Gigerenzer et al., 1991) contends that if general knowledge tasks are randomly sampled from a natural environment, overconfidence should be zero. The account developed here suggests that overconfidence depends on the relation between the strength of the evidence—how strongly it seems to point toward an answer—and its weight, including how reliable and substantial it is. Overconfidence will be most pronounced when the weight of evidence is low and the strength of evidence is high. This is likely to arise in domains where people can readily form a strong impression even though these impressions have low predictive validity—for example, an interviewer forming a strong impression of a graduate student’s mind despite those impressions failing to predict performance.

To investigate these ideas, an experiment presented subjects with 30 pairs of U.S. states randomly selected from the set of all possible pairs, drawn from the 1990 World Almanac. Subjects were asked to choose which state in each pair was higher on a given attribute and to assess the probability that their answer was correct. Three attributes were chosen to vary the strength of impression and the amount of knowledge subjects were likely to have: Population (number of people in each state), Education (high-school graduation rate), and Voting (difference in voting rates between the last two presidential elections). It was hypothesized that subjects would be most knowledgeable about Population, leading to high accuracy and high confidence. For Voting, low accuracy and low confidence were expected. For Education, people’s stereotypes or images of states (e.g., famous universities, cultural events) were expected to produce a strong sense of confidence, but because those cues have low validity for predicting graduation rates, greater overconfidence was anticipated—low accuracy coupled with higher confidence.

Each of 298 subjects evaluated 15 pairs on one attribute, indicating their confidence for each answer. Afterward, they were asked to estimate how many of the 15 questions they had answered correctly, with a reminder that chance would yield 7.5 correct.

The results showed significant overconfidence in individual probability judgments for all three attributes, contradicting the claim that random sampling from a natural environment eliminates overconfidence. For Population, accuracy was high and confidence was high; for Voting, both were relatively low; and for Education, accuracy was low while confidence remained relatively high—consistent with the hypothesis that overconfidence arises when the strength of evidence outstrips its weight.

In contrast, estimated frequencies of correct answers were substantially below actual frequencies for all three attributes. In fact, for two of the three attributes these estimates fell below chance. One possible explanation is that subjects reported the number of items they knew with certainty, without correcting for guessing. Similar results have been observed by other investigators. Evidently, people can maintain a high degree of confidence in the validity of specific answers even when they know that their overall hit rate is not very high—a statistical version of the paradoxical statement “I believe in all of my beliefs, but I believe that some of my beliefs are false.” This phenomenon has been called the illusion of validity: people often make confident predictions about individual cases on the basis of fallible data (e.g., personal interviews or projective tests), even when they know those data have low predictive validity.

The discrepancy between estimates of frequency and judgments of confidence is noteworthy, but it does not diminish the significance of overconfidence in individual items. The latter is important because people’s decisions are commonly based on their confidence in their assessment of individual events, not on estimates of their overall hit rate. For example, an extensive survey of new business owners revealed that entrepreneurs were, on average, highly optimistic (i.e., overconfident) about the success of their specific new ventures even when they were reasonably realistic about the general rate of failure for ventures of that kind. Decisions to undertake new ventures are based primarily on beliefs about individual events, rather than on overall base rates. This tendency to favor an individual or “inside” view over a statistical or “outside” view represents a major departure of intuitive judgment from normative theory.

Finally, performance on the frequency task itself left much to be desired. The degree of underestimation in judgments of frequency was, on average, comparable to the degree of overconfidence in individual probability judgments. Moreover, the correlation across subjects between estimated and actual frequency was negligible for all three attributes (+.10 for Population, −.10 for Voting, and +.15 for Education). These observations do not support the view that people correctly estimate their hit rate and that the confidence-frequency gap merely reflects an inability to evaluate the probability of unique events.