Sample Size Neglect (Strength-Weight Bias)

Table 13.1. Stimuli and Responses for Study 1

Table 13.1 also presents, for each sample of data (D), the posterior probability for hypothesis H (a 3 : 2 bias in favor of heads) computed according to Bayes’ rule. Assuming equal prior probabilities, Bayes’ rule yields

P(H|D) = \frac{n}{n + 1} \cdot \frac{h - t}{n} \cdot d

where h and t are the number of heads and tails, respectively, and n = h + t denotes sample size. The first term on the right side, n, represents the weight of evidence. The second term, the difference between the proportion of heads and tails in the sample, represents the strength of the evidence for H against T. The third term, which is held constant in this study, is the discriminability of the two hypotheses, corresponding to d in signal detection theory. Plotting equal-support lines for strength and weight in logarithmic coordinates yields a family of parallel straight lines with a slope of −1, as illustrated by the dotted lines in Fig. 13.1. [To facilitate interpretation, the strength dimension is defined as h/n, which is linearly related to (h − t)/n.] Each line connects all data sets that provide the same support for hypothesis H. For example, a sample size of 9 with 6 heads and 3 tails and a sample size of 17 with 10 heads and 7 tails yields the same posterior probability (.77) for H over T. Thus, both point (9, 6/9) and point (17, 10/17) lie on the upper line. Similarly, the lower line connects the data sets that yield a posterior probability of .60 in favor of H (see Table 13.1).

Figure 13.1. Equal support lines for strength and sample size.

To compare the observed judgments with Bayes’ rule, we first transformed each probability judgment into log odds; then, for each subject as well as the median data, regressed the logarithm of these values against the logarithms of strength, (h − t)/n, and of weight, n, separately for each subject. The regressions fit the data quite well: multiple R was .95 for the median data and .82 for the median subject. According to Bayes’ rule, the regression weights for strength and weight in this metric are equal (see Fig. 13.1). In contrast, the regression coefficient for strength was larger than the regression coefficient for weight for 30 out of 35 subjects (p < .001 by sign test). Across subjects, the median ratio of these coefficients was 2.2 to 1 in favor of strength. For the median data, the observed regression weight for strength (.81) was almost three times larger than that for weight.

The equal-support lines obtained from the regression analysis are plotted in Fig. 13.1 as solid lines. The comparison of the two sets of lines reveals two noteworthy observations. First, the intuitive lines are much shallower than the Bayesian lines, indicating that the strength of evidence dominates its weight. Second, for a given level of support (e.g., 60% or 77%), the Bayesian and the intuitive lines cross, indicating overconfidence when strength is high and weight is low, and underconfidence when strength is low and weight is high. As is seen later, the crossing point is determined primarily by the discriminability of the competing hypotheses (d).

Figure 13.2 plots the median confidence for a given sample of evidence as a function of the (Bayesian) posterior probability for two separate sample sizes. The best-fitting lines were calculated using the log odds metric. If the subjects were Bayesian, the solid lines would coincide with the dotted line. Instead, intuitive judgments based on the small sample (n = 5) were overconfident, whereas the judgments based on the larger sample (n = 17) were underconfident.

Figure 13.2. Sample size and confidence.

The results described in Table 13.1 are in general agreement with previous results that document the non-normative nature of intuitive judgment (for reviews, see, e.g., Kahneman, Slovic, & Tversky, 1982; von Winterfeld & Edwards, 1986). Moreover, they help reconcile apparently inconsistent findings. Edwards and his colleagues (e.g., Edwards, 1968), who used a sequential updating paradigm, argued that people are conservative in the sense that they do not extract enough information from sample data. However, Tversky and Kahneman (1971), who investigated the role of sample size in researchers’ confidence in the replicability of their results, concluded that people (even those trained in statistics) make radical inferences on the basis of small samples. Figures 13.1 and 13.2 suggest how the dominance of sample proportion over sample size could produce both findings. In some updating experiments conducted by Edwards, subjects were exposed to large samples of data typically of moderate strength. This is the context in which we expect underconfidence or conservatism. The situations studied by Tversky and Kahneman, however, involve moderately strong effects based on fairly small samples. This is the context in which overconfidence is likely to prevail. Both conservatism and overconfidence, therefore, can be generated by a common bias in the weighting of evidence; namely, the dominance of strength over weight.

As noted earlier, the tendency to focus on the strength of the evidence leads people to neglect or underweight other variables, such as the prior probability of the hypothesis in question or the discriminability of the competing hypotheses. These effects are demonstrated in the following two studies. All three studies reported in this section use a within-subject design, in which both the strength of the evidence and the mitigating variable (e.g., sample size) are varied within subjects. This procedure may underestimate the dominance of strength because people tend to respond to whatever variable is manipulated within a study whether or not it is normative to do so (Fischhoff & Bar-Hillel, 1984). Indeed, the neglect of sample size and base rate information has been most pronounced in between-subject comparisons (Kahneman & Tversky, 1972).