Generalizing from Instances

GENERALIZING FROM INSTANCES

Generalization from observed cases is the classic concern of philosophers and other thinkers who are interested in induction. A number of instances of Class A are observed, and each of them turns out to have Property B. Possible inferences include the universal generalization all As have B, or the near universal most As have B, or at least the relinquishing of the contrary generalization; namely, most As do not have B.

The untrammeled employment of the representativeness heuristic would lead people to make the above inferences from quite small numbers of instances, and, indeed, this is often found, both anecdotally and in laboratory studies (Nisbett & Ross, 1980, pp. 77–82). However, philosophers since Hume have puzzled about how these generalizations can be logically justified, even when very large numbers of instances are observed. The puzzle has been compounded by the fact that sometimes it seems correct to generalize confidently from a few instances. Hume (1748/1955) wrote, “[Often, when] I have found that . . . an object has always been attended with . . . an effect . . . I foresee that other objects which are in appearance similar will be attended with similar effects” (p. 48). The problem is that only sometimes do we draw such a conclusion with confidence. “Nothing so like as eggs, yet no one, on account of this appearing similarity, expects the same taste and relish in all of them” (Hume, 1748/1955, p. 50). Mill (1843/1974), a century later, phrased the problem as: “Why is a single instance, in some cases, sufficient for a complete induction, while in others myriads of concurring instances, without a single exception known or presumed, go such a very little way towards establishing a universal proposition?” (p. 314).

The statistical advances since Mill’s time make it clear that a large part of the answer to his question has to do with beliefs about the variability or homogeneity of certain kinds or classes of events (cf. Thagard & Nisbett, 1982). Generalization from a large sample is justified in terms of one’s beliefs that the sampling itself is homogeneous (i.e., that the distribution of possible sample statistics is the same as would be predicted by random sampling). Generalization from a small sample or resistance to generalization, even from a large sample, are justified in terms of prior beliefs about the homogeneity or heterogeneity of objects or events of a certain kind with respect to a property of a certain kind.

Homogeneity Study: Beliefs About Homogeneity and Reliance on the Law of Large Numbers

In this study, we simply guessed at the prevailing beliefs about homogeneity. We tried to obtain different degrees of heterogeneity by using conductivity of metals, colors of animals, and so on. Subjects were told of one instance or of several instances of a sampled object having a particular property and were asked to guess what percentage of the population of all such objects would have the property. The sample sizes used were 1, 3, or 20; in the latter cases, all 3 or all 20 of the objects had the property in question. We anticipated that subjects would generalize more readily from a given number of instances when the kind of object was perceived as homogeneous with respect to the kind of property than when the kind of object was perceived as heterogeneous with respect to the kind of property.

Method. Subjects were 46 University of Michigan students of both sexes who were enrolled in introductory psychology. (As sex did not affect any of the dependent variables in this or any of the other studies, it is not discussed further.) Eighty-five percent of the subjects had taken no statistics courses in college. The questionnaire was presented as one of several in a study on judgment. It read as follows for the N = 1 condition:

Imagine that you are an explorer who has landed on a little known island in the Southeastern Pacific. You encounter several new animals, people, and objects. You observe the properties of your “samples” and you need to make guesses about how common these properties would be in other animals, people or objects of the same type.

Suppose you encounter a new bird, the shreeble. It is blue in color. What percent of all shreebles on the island do you expect to be blue?

(This and the subsequent questions were followed by “__________percent. Why did you guess this percent?”)

Suppose the shreeble you encounter is found to nest in a eucalyptus tree, a type of tree which is fairly common on the island. What percent of all shreebles on the island do you expect to nest in eucalyptus trees?

Suppose you encounter a native, who is a member of a tribe he calls the Barratos. He is obese. What percent of the male Barratos do you expect to be obese?

Suppose the Barratos man is brown in color. What percent of male Barratos do you expect to be brown (as opposed to red, yellow, black, or white)?

Suppose you encounter what the physicist on your expedition describes as an extremely rare element called floridium. Upon being heated to a very high temperature, it burns with a green flame. What percent of all samples of floridium found on the island do you expect to burn with a green flame?

Suppose the sample of floridium, when drawn into a filament, is found to conduct electricity. What percent of all samples of floridium found on the island do you expect to conduct electricity?

The questionnaires for the N = 3 condition and the N = 20 condition were identical except that they specified larger samples of each object. For example, the first shreeble item for the N = 3 condition read as follows:

Suppose you encounter a new bird, the shreeble. You see three such birds. They are all blue in color. What percent of all shreebles on the island do you expect to be blue?

The reasons subjects gave for guessing as they did were coded as to their content. There were three basic sorts of answers: (1) references to the homogeneity of the kind of object with respect to the kind of property; (2) references to the heterogeneity of the kind of object with respect to the kind of property – due to the different properties of subkinds (e.g., male versus female), to some causal mechanism producing different properties (e.g., genetic mistakes), or to purely statistical variability (e.g., “Where birds nest is sometimes just a matter of chance”); and (3) other sorts of answers that were mostly based on representativeness or that were mere tautologies. Two independent coders achieved 89% exact agreement on coding category.

Results. Any one element is presumed by scientists to be homogeneous with respect to most properties. At the other extreme, most human groups are highly heterogeneous among themselves in many attributes, including body weight. If educated laypeople share these beliefs and if they reason statistically, then (1) they should exercise more caution in generalizing from single cases when heterogeneity is expected than when homogeneity is expected and (2) large N should be important primarily in the case of populations whom subjects believe to be heterogeneous with respect to the property in question.

Figure 28.1 presents subjects’ estimates of the percentage of each population having the property associated with the sample as a function of sample size presented. It may be seen that subjects are quite willing to generalize from even a single instance of green-burning or electricity-conducting floridium and also from a single, brown, Barratos tribesman. The modal estimate for N = 1 (as well as for N = 3 and N = 20) in all of these cases is 100%. In contrast, generalizations are less extreme for even 20 instances of blue shreebles or eucalyptus-nesting shreebles or 20 obese Barratos. The t (31) contrasting N = 1 for floridium attributes and Barratos color with N = 20 for shreeble attributes and Barratos obesity is 3.00; p < .01.

Figure 28.1. Percentage of each population estimated to have the sample property as a function of number of cases in the sample.

Subjects’ explanations for their estimates fully justify this pattern of inferences. It may be seen in Table 28.1 that subjects reported believing that elements are homogeneous with respect to color and conductivity and that tribes are homogeneous with respect to color. In contrast, subjects rarely expressed the belief that there is homogeneity for the other kinds of populations and properties and instead expressed belief in heterogeneity of one sort or another for these objects and properties. Figure 28.1 shows that it is only for these latter cases that subjects reasoned statistically in the sense that they were more willing to assume that the population resembles the sample when N is larger. N affects the estimates of the obesity of Barratos and the color of shreebles (p < .001 and p = .11, respectively).