Limitations of Half-Range Studies in Calibration Research

Out of the Laboratory

Most theories of judgmental calibration were developed on the basis of laboratory data from the two-alternative (half-range) forced-choice (2AFC) paradigm, usually with general knowledge questions as the object of judgment (see Lichtenstein et al., 1982). There are a number of reasons for this tradition. General knowledge or “almanac” studies are quick and efficient, producing hundreds of easy-to-score judgments from hundreds of subjects, whereas it is both difficult and time-consuming to assess the quality of judgments made in applied settings: Loftus and Wagenaar (1988), for example, took 3 years to collect judgments about self-reported wins and losses from 182 trial lawyers. The two-alternative paradigm is closely linked to research and theory on “metacognition,” the study of how well people can monitor their own knowledge (e.g., Nelson, Leonesio, Landwehr, & Narens, 1986), and more generally has a long history in psychophysics. In addition, the half-range measure of probability fits the intuitive notion of “confidence,” running from complete uncertainty (.5) to complete certainty (1.0).

This reliance on half-range studies, however, has obscured several important issues. For full-range judgments the focal hypothesis for a given domain can be defined in alternate ways (e.g., a set of basketball games can be assessed in terms of wins for the home team, or wins for the team higher in the standings), and many measures of judgmental calibration are sensitive to the choice of the designated event (Liberman & Tversky, 1993). This complication is avoided in half-range studies, but at the cost of losing some important information about human judgment. Most critically, half-range studies confound the role of outcome base rate and task difficulty. In a typical half-range study, the outcome can be summarized by the overall percent correct, which serves to define the expected outcome or base rate as well as the difficulty of the question set.

In a half-range study, the effects of base rate and difficulty or discriminability may be distinguished by designating one of the possible responses to each item as a target, implicitly creating a one-alternative judgment task (cf. Ferrell & McGoey, 1980). However, these manipulated targets are often arbitrary [e.g., P(True) versus P(False)]. In true one-alternative judgments (e.g., predicting the likelihood of a given disease, predicting the likelihood of winning a court case), the base rate of the positive outcome is clearly separable from the difficulty of discriminating positive outcomes from negative outcomes. Predicting a nuclear accident or a stock market bubble may be difficult either because the event is rare or because the available evidence is not diagnostic (or both) – a distinction that may make all the difference to understanding and debiasing such judgments.