Expert Judgment: A Question of Competence?

Expert Judgment: A Question of Competence?

Many commentators have argued that expert judgments, at least those made in a naturalistic setting, may be immune from the systematic judgmental biases found in laboratory experiments. The prevalence of judgmental biases is often blamed on factors intrinsic to the psychology laboratory such as unmotivated subjects and misleading and unrepresentative questions (see, e.g., Shanteau, 1999). Thus, “decision heuristics . . . appear more likely to create biases in the psychology laboratory than in the [medical] clinic” (Schwarz & Griffin, 1986, p. 82), and “biases found readily in other research are not evident in the judgements of professional auditors” (Smith & Kida, 1991, p. 485). By and large such conclusions are based on studies of domain experts completing pencil-and-paper measures in their domains of expertise, rather than from the observation and assessment of expert judges in their day-to-day activities.

There are many reasons to expect that the judgments of experts will depart systematically from the judgments of novices (Shanteau, 1999). Experts know more, they can organize cues into larger “chunks” (Chase & Simon, 1973), and they can recognize patterns more easily, more accurately and more quickly (Glaser, 1990; Klein, 1998). As Simon noted in his classic studies of expert judgment, experts seem particularly good at that part of recognition that consists of automatic pattern matching between a stimulus and a category (Chase and Simon, 1973; Simon, 1979). All these statements are consistent with a direct support account, and in terms of our guiding model imply that in well-ordered domains, expert judgments will have a higher α than novice judgments, reflecting experts’ greater ability to correctly discriminate between different outcomes. However, the ability to order cases in terms of the likelihood of a given outcome (discrimination) does not necessarily correspond to the ability to assign the appropriate level of probability to that outcome (calibration) (Liberman & Tversky, 1993). Thus, depending on the problem environment, even the most knowledgeable and perceptive expert may nonetheless show the patterns of miscalibration portrayed in Fig. 39.1 (see Stewart, Roebber, & Bosart, 1997, and Wagenaar & Keren, 1986, for related arguments).