It's Not Always Wrong to Be Wrong

IT’S NOT ALWAYS WRONG TO BE WRONG

In an alternate universe, one where later research on tobacco came out differently, we might have found that the statistician R. A. Fisher’s odd-sounding theory was right after all, and smoking was a consequence of cancer instead of the other way around. It wouldn’t be the biggest reversal medical science has ever suffered, by a long shot. And what then? The surgeon general would have issued a press release saying, “Sorry, everyone can go back to smoking now.” In the interim, tobacco companies would have lost a lot of money, and millions of smokers would have forgone billions of pleasurable cigarettes. All because the surgeon general had declared as a fact what was only a strongly supported hypothesis.

But what was the alternative? Imagine what you’d have to do in order to really know, with something like absolute assurance, that smoking causes lung cancer. You’d have to collect a large population of teenagers, select half of them at random, and force that half to spend the next fifty years smoking cigarettes on a regular schedule, while the other half would be required to abstain. Jerry Cornfield, an early pioneer of smoking research, called such an experiment “ possible to conceive but impossible to conduct.” Even if such an experiment were logistically possible, it would violate every ethical norm in existence about research on human subjects.

Makers of public policy don’t have the luxury of uncertainty that scientists do. They have to form their best guesses and make decisions on the basis thereof. When the system works—as it unquestionably did in the case of tobacco—the scientist and the policy maker work in concert, the scientist reckoning how uncertain we ought to be and the policy maker deciding how to act under the uncertainty thus specified.

Sometimes this leads to mistakes. Hormone replacement therapy was long thought to protect postmenopausal women against heart disease, based on observed correlations. Later randomized experiments challenged that claimed protection.

In 1976 and again in 2009, the U.S. government embarked on massive and expensive vaccination campaigns against the swine flu, having received warnings from epidemiologists each time that the currently prevailing strain was particularly likely to go catastrophically pandemic. In fact, both flus, while severe, fell well short of disastrous.

It’s easy to criticize the policy makers in these scenarios for letting their decision making get ahead of the science. But it’s not that simple. It’s not always wrong to be wrong.

How can this be so? A quick expected value computation helps unpack the seemingly paradoxical slogan. Suppose we’re considering making a health recommendation—say, that people should stop eating eggplant because eggplant induces a small risk of sudden catastrophic heart failure. This conclusion is based on a series of studies that found eggplant eaters slightly more likely than non−eggplant eaters to keel over dead without warning. But there’s no prospect of doing a large-scale randomized controlled trial where we force eggplants on some people and deny them to others. We have to make do with the information we have, which represents a correlation only. For all we know, there’s a common genetic basis for eggplantophilia and cardiac arrest. There’s no way to be sure.

Perhaps we are 75% sure that our conclusion is correct and that a campaign against eggplant would save a thousand American lives per year. But there’s also a 25% chance our conclusion is wrong; and if it’s wrong, we’ve induced many people to give up what might be a favorite vegetable, leading them to eat a less healthy diet overall, and causing, let’s say, two hundred excess deaths annually.*

As always, we obtain the expected value by multiplying the result of each possible outcome by the corresponding probability, and then adding everything up. In this case, we find that

75% × 1000 + 25% × (−200)= 750 − 50= 700.

So our recommendation has an expected value of seven hundred lives saved per year. Over the loud and well-financed complaints of the Eggplant Council, and despite our very real uncertainty, we go public.

Remember: the expected value doesn’t represent what we literally expect to happen, but rather what we might expect to happen on average were the same decision to be repeated again and again. A particular public health decision cannot simply be repeated like a coin flip. On the other hand, eggplants are not the only environmental danger we may be called upon to assess. Maybe it will come to our attention next that cauliflower is associated with arthritis, or vibrating toothbrushes with autism. If each intervention has an expected value of seven hundred lives a year, that is the estimated average benefit across such decisions. In any individual case, we might do more harm than good. The argument for acting depends on the quality of our estimates and on whether we have accounted for the relevant harms and alternatives.

And if we held ourselves to a stricter evidentiary standard, declining to issue any of these recommendations because we weren’t sure we were right? Then the lives we would have saved would be lost instead.

It would be great if we could assign precise, objective probabilities to real-life health conundrums, but of course we can’t. This is another way that the interaction of a drug with the human body differs from a coin you can flip or a lottery ticket you can scratch. We’re stuck with the messy, vague probabilities that reflect our degree of belief in various hypotheses, the probabilities that R. A. Fisher loudly denied were probabilities at all. So we don’t and can’t know the exact expected value of launching a campaign against eggplant or vibrating toothbrushes, or tobacco. But often we can say with confidence that the expected value is positive. That doesn’t mean a particular campaign is sure to have good effects. It means we estimate that its benefits outweigh its harms. We can be mistaken about either. Refusing to make any recommendation that might be wrong has costs too: it rules out actions that could prevent substantial harm. It’s a lot like George Stigler’s advice about missing planes. If you never give advice until you’re sure it’s right, you’re not giving enough advice.