Pattern Noise and the Ambiguity of Scales

F IGURE 14:

Noise in value vs. noise in ranking

An Unfortunate Conclusion

The results are consistent with the theory we have outlined: dollar awards for all cases were anchored on the arbitrary number that each juror picked for the first case they saw. The relative ranking of cases reflects attitudes with fair accuracy and is thus not very noisy, but the absolute values of the dollar awards are essentially meaningless because they depend on the arbitrary number chosen in the first case.

Ironically, the case that jurors assess in real trials is the first and only one they see. American legal practice requires civil juries to set a dollar award for one case, without the benefit of any guiding anchor. The law explicitly prohibits any communication to the jury of the size of punitive awards in other cases. The assumption implicit in the law is that jurors’ sense of justice will lead them directly from a consideration of an offense to the correct punishment. This assumption is psychological nonsense—it assumes an ability that humans do not have. The institutions of justice should acknowledge the limitations of the people who administer it.

The example of punitive damages is extreme; professional judgments are rarely expressed on scales that are so hopelessly ambiguous. Nonetheless, ambiguous scales are common, which means that the punitive-damages study holds two general lessons, applicable in business, education, sports, government, and elsewhere. First, the choice of a scale can make a large difference in the amount of noise in judgments, because ambiguous scales are noisy. Second, replacing absolute judgments with relative ones, when feasible, is likely to reduce noise.

Speaking of Scales

“There is a lot of noise in our judgments. Could this be because we understand the scale differently?”

“Can we agree on an anchor case that will serve as a reference point on the scale?”

“To reduce noise, maybe we should replace our judgments with a ranking?”

CHAPTER 16

Patterns

R emember Julie, the precocious child whose college GPA you tried to guess in chapter 14 ? Here is a fuller description.

Julie was an only child. Her father was a successful lawyer, her mother an architect. When Julie was about three years old, her father contracted an autoimmune disease that forced him to work at home. He spent a lot of time with Julie and patiently taught her to read. She was reading fluently when she was four years old. Her dad also tried to teach her arithmetic, but she found that topic difficult. Julie was a good pupil in elementary school, but she was emotionally needy and rather unpopular. She spent much time alone and became a passionate bird-watcher after being inspired by watching birds with her favorite uncle.

Her parents divorced when she was eleven, and Julie took the divorce hard. Her grades plummeted, and she had frequent outbursts at school. In high school, she did very well in some subjects, including biology and creative writing. She surprised everyone by excelling in physics. But she neglected most of her other subjects, and she graduated from high school a B student.

Not admitted to the prestigious schools

to which she applied, Julie eventually attended a good state school, where she majored in environmental studies. During her first two years in college, she continued a pattern of frequent emotional entanglements and smoked pot fairly regularly. In her fourth semester, however, she developed a strong wish to go to medical school and began to take her work much more seriously.

What is your best guess about Julie’s graduating GPA?

Problems: Hard and Easy

Obviously, this problem (let’s call it Julie 2.0) has become much more difficult. All you knew about Julie 1.0 was that she could read when she was four. With only one cue, the power of matching did the work, and an intuitive estimate of her GPA came quickly to mind.

Matching would still be available if you had several cues that pointed in the same general direction. For instance, when you read a description of Bill, the jazz-playing accountant, all the information you had (“unimaginative,” “strong in mathematics,” “weak in social studies”) painted a coherent, stereotypical picture. Similarly, if most of the events in the life of Julie 2.0 were consistent with a story of precociousness and superior achievement (with perhaps a few data points suggesting merely “average” performance), you would not find the task so difficult. When the evidence available paints a coherent picture, our fast, System 1 thinking has no difficulty making sense of it. Simple judgment problems like these are easily resolved, and most people agree on their solution.

Not so with Julie 2.0. What makes this problem difficult is the presence of multiple, conflicting cues. There are indications of ability and motivation but also of character weaknesses and mediocre achievement. The story seems to be all over the place. It does not easily make sense, because the elements cannot be fit in a coherent interpretation. Of course, the incoherence does not make the story unrealistic or even implausible. Life is often more complex than the stories we like to tell about it.

Multiple, conflicting cues create the ambiguity that defines difficult judgment problems. Ambiguity also explains why complex problems are noisier than simple ones. The rule is simple: if there is more than one way to see anything, people will vary in how they see it. People can pick different pieces of evidence to form the core of their narrative, so there are many possible conclusions. If you found it difficult to construct a story that makes sense of Julie 2.0, you can be quite certain that other readers will construct different stories that justify judgments other than yours. This is the variability that produces pattern noise.

When do you feel confident in a judgment? Two conditions must be satisfied: the story you believe must be comprehensively coherent, and there must be no attractive alternatives. Comprehensive coherence is achieved when all the details of the chosen interpretation fit with the story and reinforce each other. Of course, you can also achieve coherence, albeit less elegantly, by ignoring or explaining away whatever does not fit. It is the same with alternative interpretations. The true expert who has “solved” a judgment problem knows not only why her explanatory story is correct; she is equally fluent in explaining why other stories are wrong. Here again, a person can gain confidence of equal strength but poorer quality by failing to consider alternatives or by actively suppressing them.

The main implication of this view of confidence is that subjective confidence in one’s judgment by no means guarantees accuracy. Moreover, the suppression of alternative interpretations—a well-documented process in perception—could induce what we have called the illusion of agreement (see chapter 2 ). If people cannot imagine possible alternatives to their conclusions, they will naturally assume that other observers must reach the same conclusion, too. Of course, few of us have the good fortune of being highly confident about all our judgments, and all of us have had the experience of uncertainty, perhaps as recently as your reading about Julie 2.0. We are not all highly confident all the time, but most of the time we are more confident than we should be.

Pattern Noise: Stable or Transient

We have defined a pattern error as an error in an individual’s judgment of a case that cannot be explained by the sum of the separate effects of the case and the judge. An extreme example may be the normally lenient judge who is unusually severe in sentencing a particular kind of defendant (say, people who have committed traffic offenses). Or, say, the normally cautious investor who drops his usual caution when shown the plan of an exciting start-up. Of course, most pattern errors are not extreme: we observe a moderate pattern error in the lenient judge who is less lenient than usual when dealing with recidivists, or even more lenient than usual when sentencing young women.

Pattern errors arise from a combination of transient and permanent factors. The transient factors include those we have described as sources of occasion noise, such as a judge’s good mood at the relevant moment or some unfortunate recent occurrence that is currently on the judge’s mind. Other factors are more permanent—for example, an employer’s unusual enthusiasm for people who attended certain universities or a doctor’s unusual propensity to recommend hospitalization for people with pneumonia. We can write a simple equation that describes an error in a single judgment:

Pattern Error = Stable Pattern Error + Transient (Occasion) Error

Because stable pattern error and transient (occasion) error are independent and uncorrelated, we can extend the equation above to analyze their variances:

(Pattern Noise) 2 = (Stable Pattern Noise) 2

• (Occasion Noise) 2

As we did for other components of error and noise, we can represent this equation graphically as a sum of squares on the sides of a right triangle (Figure 15 ):