Sample Size Neglect

Manipulating the Salience of Chance Factors

Making chance factors more salient may influence how people generalize from evidence. One potentially interesting way of doing this would be to highlight for subjects the degree to which evidence about an object should properly be regarded as a sample from the population of the object’s attributes. Such a reminder ought to prompt subjects to reason more statistically, deemphasizing evidence from smaller samples and placing greater weight on evidence from larger samples.

Borgida and Nisbett (1977) argued that people often ignore the judgments of others when choosing between two objects and substitute their own initial impressions of the objects as the sole basis of choice. People do this in part because they do not recognize the relevance of the law of large numbers when reasoning about events of the personal preference kind. When the objects are multifaceted and complex, however, the law of large numbers is applicable in two ways: (1) The reactions of other people to the object, especially if they are based on more extensive contact with the object than one has had oneself, generally should be a useful guide to choice (although, of course, it is possible to construct cases in which other people’s reactions would not be useful); and (2) One’s own experience with the object, especially if it is brief or superficial, may be a poor guide to choice because of the error that plagues any small samples, even those that happen to be our own.

It seemed likely that if people were made explicitly aware of the role of chance in determining the impression one may get from a small sample, they might place less faith in a small personal sample and more faith in a large sample based on other people’s reactions.

Method. Subjects were 157 University of Michigan students of both sexes who were enrolled in introductory psychology classes. Eighty-seven percent had taken no statistics courses in college. Subjects participated in small groups. They were presented with two versions of the following problem.

David L. was a senior in high school on the East Coast who was planning to go to college. He had compiled an excellent record in high school and had been admitted to his two top choices: a small liberal arts college and an Ivy League university. David had several older friends who were attending the liberal arts college and several who were attending the Ivy League university. They were all excellent students like himself and had interests similar to his. The friends at the liberal arts college all reported that they liked the place very much and that they found it very stimulating. The friends at the Ivy League university reported that they had many complaints on both personal and social grounds and on educational grounds.

David initially thought that he would go to the smaller college. However, he decided to visit both schools himself for a day.

He did not like what he saw at the private liberal arts college: Several people whom he met seemed cold and unpleasant, a professor he met with briefly seemed abrupt and uninterested in him, and he did not like the “feel” of the campus. He did like what he saw at the Ivy League university: Several of the people he met seemed like vital, enthusiastic, pleasant people; he met with two different professors, who took a personal interest in him; and he came away with a very pleasant feeling about the campus. Please say which school you think David should go to.

1. He should definitely go to the liberal arts college.

2. He should probably go to the liberal arts college.

3. It’s a toss-up.

4. He should probably go to the Ivy League university.

5. He should definitely go to the Ivy League university.

Please indicate why you made the recommendation you did.

The remaining subjects were presented with an identical problem except that the possibilities for error in David L.’s sample were highlighted by having him draw up a list of all the things that might be seen on the two campuses and then selecting randomly from among them for his actual schedule. The following was added to the second paragraph of the no-cue version.

He proceeded systematically to draw up a long list for both colleges of all the classes which might interest him and all the places and activities on campus that he wanted to see. From each list, he randomly selected several classes and activities to visit, and several spots to look at (by blindly dropping a pencil on each list of alternatives and seeing where the point landed).

Open-ended responses to the probe question were coded (by a blind coder) as to whether subjects justified their choice by showing any recognition of the statistical questions involved – either the dubiousness of David L.’s impressions because of the quantity of his evidence or the superiority of his friends’ testimony because of their greater experience. There was 90% agreement among coders as to the assignment of an answer to the statistical versus nonstatistical categories.

Results. When there was no sampling cue pointing to the probabilistic nature of David L.’s personal evidence base, 74% of the subjects recommended that David L. should go to the Ivy League university, which his friends did not much like but where he enjoyed his day. When the sampling cue was present, this dropped to 56% (χ2 = 5.38, p *< .*025). Moreover, subjects in the probabilistic-cue condition were much more likely to refer to statistical considerations having to do with the adequacy of the sample. Fifty-six percent of probabilistic-cue subjects raised statistical questions in their open-ended answers, whereas only 35% of subjects in the no-cue condition did so (p *< .*01). Thus, when subjects are prompted to consider the possibilities for error that are inherent in a small sample of events, they are likely to shift to a preference for large indirect samples over small personal ones, and their open-ended answers suggest that statistical considerations contribute to this shift.

The findings of Study 3 are extremely ironic in that subjects are more likely to reject the superior personal evidence in the probabilistic-cue condition than to reject the inferior personal evidence in the control condition. This is because the same circumstances that serve to make the evidence superior in the probabilistic-cue condition also serve to make salient the extreme heterogeneity of the event population to be estimated and the small size of the personal sample of those events. It is important to note that when Study 3 is run with a “within” design, subjects assess the relative value of the personal evidence correctly; that is, they generally rate the quality of evidence in the probabilistic-cue condition as superior to that in the control condition. In two slightly different within-design follow-ups to Study 3, subjects rated the probabilistic-cue sample as being superior to the sample in the control version. In one of the follow-ups (where subjects read the control problem and rated the quality of the personal evidence, then read the cue paragraph and compared the quality of the evidence there with the control version) four times as many subjects preferred the probabilistic-cue evidence as preferred the control evidence. In the other follow-up (where subjects actually acted as subjects in the control condition and then were shown the cue version), 40% more subjects preferred the probabilistic-cue evidence than preferred the control evidence.