Effects of Statistical Training on Everyday Reasoning

THE EFFECTS OF TRAINING IN STATISTICS ON REASONING ABOUT EVERYDAY EVENTS

Perhaps the most important implication of the present view is that statistical reasoning about everyday events should be highly trainable. A major reason for optimism is that, people’s intuitive reasoning skills already include strategies that may be called statistical heuristics. Formal training in statistics, therefore, should represent less a grafting on of procedures than a refinement of preexisting ones. Formal training seems likely to improve reasoning for three distinct, but mutually supportive, reasons.

1. Training in statistics is apt to facilitate the recognition of event distributions and their statistical parameters. It can be difficult to apply rules such as the law of large numbers unless the units of evidence can be identified and, hence, the sample space and distribution of the event units. It seems likely that training in statistics could provide quite general skills in construing evidence in such a way that it can be properly unitized, the sample space identified, and parameters recognized. Training in probability theory, especially in permutations and combinations, should be particularly likely to be helpful in this regard as should test theory, which requires the student to recognize, for example, the different reliability of tests composed of different numbers of units – items, trials, occasions, and so on. But ordinary inferential statistics also introduces a fair amount of instruction in unitizing evidence: In order to measure the corn yield of a type of seed, for example, it is necessary to measure the yield for some unit of sampling (e.g., individual plot, individual farm). Statistics courses should also make it easier for people to think usefully about parameters of distributions – about central tendencies and about dispersion.

2. Training in statistics is apt to facilitate the recognition of the role of chance in producing events. A major concept of parametric statistics is that of error. Every inferential test features an estimate of error, and much of statistical training centers on questions of allocation of effects to the systematic category versus the random category. It seems possible that the focus on the concept of error in statistics might heighten the salience of uncertainty in daily life. A second major concept that might be helpful in recognizing uncertainty is the formal notion of sampling. This might be of general use in construing evidence as a (possibly small and inaccurate) sample from a (possibly heterogeneous) population.

3. Training in statistics is apt to improve the clarity and accessibility of statistical rules and should expand the repertoire of statistical rules. In effect, statistics training should hone intuitive heuristics into more precise tools. It seems clear that without training some statistical rules are poorly understood at any level of abstraction and in any context. Rules of covariation assessment and some versions of the regression principle are particularly difficult and may not even be represented in most people’s intuitive repertoire. But even relatively intuitive rules such as the law of large numbers have nearly limitless corollaries and implications, some of which may be much easier to discern with formal training. Although people understand the “law of small large numbers,” they may not be able to extend the principle to numbers of nonexperiential magnitudes without formal training. It may be the lack of a fully formal understanding of the law of large numbers that prevents people from applying it in the maternity ward problem, for example, where the sample sizes involved (15 and 45) are not often represented in people’s everyday experience of sampling and variability.

The evidence indicates that statistical training does indeed have profound effects on people’s reasoning about everyday life events. In one series of studies, Krantz, Fong, and Nisbett (1983) examined four groups of subjects differing widely in educational level. Subjects, who were college students with or without statistical training, graduate students with a fair amount of statistical training, or PhD level scientists with several years of training, were presented with one of a pair of restaurant problems. In each problem, a protagonist experienced a truly outstanding meal on the first visit to a restaurant but was disappointed on a repeat visit. The subjects were asked to explain, in writing, why this might have happened. A subject’s explanation was classified as nonstatistical if it assumed that the initial good experience was a reliable indicator that the restaurant was truly outstanding and attributed the later disappointment to a definite cause, such as a permanent or temporary change in the restaurant (e.g., “Maybe the chef quit”) or a change in the protagonist’s expectation or mood. The explanation was classified as statistical if it suggested that meal quality on any single visit might not be a reliable indicator of the restaurant’s overall quality (e.g., “Very few restaurants have only excellent meals; odds are she was just lucky the first time”). Statistical explanations were coded as to how articulate they were in indicating that a single visit may be regarded as a small sample and, hence, as unreliable. Explanations were thus coded as falling into one of three categories: (1) nonstatistical, (2) poor statistical, and (3) good statistical. The frequencies in each of these categories were used to define two dependent variables: frequency of statistical answers, defined as the proportion of responses in categories 2 and 3, and quality of statistical answers, defined as the proportion of categories 2 and 3 answers that were Category 3.

The two versions of the restaurant problem differed. The probabilistic-cue version included a random mechanism for selection from the menu: The protagonist did not know how to read a Japanese menu and selected a meal by blindly dropping a pencil on the menu and observing where the point lay. The other version had no such cue. Within each group tested, half of the subjects received the cue and half did not.

The effects of training on both dependent measures were dramatic. College students without statistical training almost never gave an answer that was at all statistical unless the problem contained the probabilistic cue, in which case about half of the answers were statistical. In contrast, more than 80% of the answers of PhD-level scientists were statistical, whether or not there was a cue. Quality of statistical answers also depended on level of training. Only 10% of the statistical answers by untrained college students were rated as good, whereas almost 80% of the statistical answers by PhD-level scientists were rated as good. It is interesting that although the presence of the probabilistic cue was very important in determining whether less trained subjects would give a statistical answer, it did not affect the quality of statistical answers for subjects at any level of training. Apparently probabilistic cues can trigger the use of statistical heuristics, but they do not necessarily improve the quality of answers: the appropriate skills must be in the individual’s repertoire to insure good quality.

The preceding study confounds training and native mathematical ability. Subsequent studies compared students at different points in the same course and tested brief training procedures, providing further evidence about whether statistical instruction influences reasoning outside the classroom and laboratory. Krantz et al. (1983) conducted a telephone “survey of opinions about sports.” Subjects were males who were enrolled in an introductory statistics course and who admitted to being at least somewhat knowledgeable about sports. One hundred subjects were randomly selected and surveyed during the first 2 weeks of the term they were enrolled in statistics. Another 93 students were surveyed at or near the end of the term. In addition to filler questions on NCAA rules and NBA salaries, subjects were asked questions for which a statistical approach was relevant, as in the example below.

In general, the major league baseball player who wins Rookie of the Year does not perform as well in his second year. This is clear in major league baseball in the past 10 years. In the American League, 8 rookies of the year have done worse in their second year; only 2 have done better. In the National League, the rookie of the year has done worse the second year 9 times out of 10.

Why do you suppose the rookie of the year tends not to do as well his second year?

Most subjects answered this question in a nonstatistical way, invoking notions like “too much press attention” and “slacking off.” Some subjects answered the question statistically (e.g., “There are bound to be some rookies who have an exceptional season; it may not be due to any great talent advantage that one guy has over the others – he just got a good year”).

The statistics course increased the percentage of statistical answers and also increased the quality of statistical answers. The course also markedly influenced both the frequency and the quality of statistical answers to another question asking subjects to explain why .450 batting averages are common the first 2 weeks of the baseball season but are unheard of as a season average. In all, the course had a significant effect on three of the five questions asked.

Finally, Fong, Krantz, and Nisbett (1983) showed that even a very brief training procedure can suffice to affect markedly subjects’ answers to problems about everyday events. There were two major elements in their training package: One covered formal aspects of sampling and the law of large numbers and the other showed how to use sampling notions as a heuristic device in modeling problems. In the sampling instruction, subjects received definitions of population and sample distributions, a statement of the law of large numbers, and a demonstration (by drawing colored gumballs from a glass vase) that a population distribution is estimated more accurately, on the average, from larger samples. The modeling, or mapping, instruction consisted of three problems (in the general style of the restaurant problem and similar to the subsequent test problems), each followed by a written solution that used the law of large numbers and emphasized the analogy between amount of evidence and size of sample.

There were four major conditions: a control group given no instruction and three experimental groups – one given sampling training only, one given mapping training only, and one given both types of training. The subjects were adults and high school students. The test consisted of 15 problems: 5 had clear probabilistic cues, 5 dealt with objective attributes such as abilities or achievements, and 5 dealt with subjective judgments. Training effects were marked for all three problem types and for both the frequency of statistical answers and the quality of statistical answers. Sampling training and mapping training were about equally effective, and in combination they were substantially more effective than either was alone. A particularly encouraging finding is that training showed no domain specificity effects. In a companion study, Fong et al. (1983) showed that it made no difference to performance whether mapping training had been on probabilistic-cue problems, objective attribute problems, or subjective judgment problems. The latter finding suggests that training on specific problem types can be readily abstracted to a degree sufficient for use on widely different problem types.

The work on training should not be taken to indicate that a statistical education is sufficient to guarantee that people will avoid errors in inductive reasoning. Kahneman and Tversky (1982; Tversky & Kahneman, 1971, 1983) have shown repeatedly that statistical expertise provides no such guarantee against errors. However, it should also be noted that courses in statistics do not emphasize ways to use statistical principles in everyday life. Were they to do so, one might see much larger differences between the educated and the uneducated than we have found.