The Insignificance of Significance

THE INSIGNIFICANCE OF SIGNIFICANCE

What’s wrong with significance? To start with, there’s the word itself. Mathematics has a funny relationship with the English language. Mathematical research articles, sometimes to the surprise of outsiders, are not predominantly composed of numerals and symbols; math is made of words. But the objects we refer to are often entities uncontemplated by the editors at Merriam-Webster. New things require new vocabulary. There are two ways to go. You can cut new words from fresh cloth, as we do when we speak of cohomology, syzygies, monodromy, and so on; this has the effect of making our work look forbidding and unapproachable. More commonly, we adapt existing words for our own purposes, based on some perceived resemblance between the mathematical object to be described and a thing in the so-called real world. So a “group,” to a mathematician, is indeed a group of things, but a very special kind of group, like the group of whole numbers or the group of symmetries of a geometric figure; we mean by it not just an arbitrary collection of things, like OPEC or ABBA, but rather a collection of things with the property that any pair of them can be combined into a third, as a pair of numbers can be added, or a pair of symmetries can be carried out one after the other.* So too for schemes, bundles, rings, and stacks, mathematical objects which stand in only the most tenuous relation to the ordinary things referred to by those words. Sometimes the language we choose has a pastoral flavor: modern algebraic geometry, for instance, is largely concerned with fields, sheaves, kernels, and stalks. Other times it’s more aggressive—it is not at all unusual to speak of an operator killing something, or, for a little more va-voom, annihilating it. I once had an uneasy moment with a colleague in an airport when he made the remark, unexceptional in a mathematical context, that it might be necessary to blow up the plane at one point.

So: significance. In common language it means something like “important” or “meaningful.” But the significance test that scientists use doesn’t measure importance. When we’re testing the effect of a new drug, the null hypothesis is that there is no effect at all; rejecting that null is evidence against it under the test’s assumptions, rather than a measure of how large the drug’s effect is. But the effect could still be very small—so small that the drug isn’t effective in any sense that an ordinary non-mathematical Anglophone would call significant.

The lexical double booking of “significance” has consequences beyond making scientific papers hard to read. On October 18, 1995, the UK Committee on Safety of Medicines (CSM) issued a “Dear Doctor” letter to nearly 200,000 doctors and public health workers around Great Britain, with an alarming warning about certain brands of “third-generation” oral contraceptives. “ New evidence has become available,” the letter read, “indicating that the chance of a thrombosis occurring in a vein is increased around two-fold for some types of pill compared with others.” A venous thrombosis is no joke; it means a clot is impeding the flow of the blood through the vein. If the clot breaks free, the bloodstream can carry it all the way to your lung, where, under its new identity as a pulmonary embolism, it can kill you.

The Dear Doctor letter was quick to assure readers that oral contraception was safe for most women, and no one should stop taking the pill without medical advice. But details like that are easy to lose when the top-line message is “Pills kill.” The AP story that ran October 19 led with “ The government warned Thursday that a new type of birth control pill used by 1.5 million British women may cause blood clots. . . . It considered withdrawing the pills but decided not to, partly because some women cannot tolerate any other kind of pills.”

The public, understandably, freaked out. One general practitioner found that 12% of pill users among her patients stopped taking their contraceptives as soon as they heard the government report. Presumably, many women switched to other versions of the pill not implicated in thrombosis, but any interruption makes the pill less effective. And less-effective birth control means more pregnancies. (What—you thought I was going to say there was a wave of abstinence?) After several successive years of decline, the conception rate in the United Kingdom jumped several percentage points the next year. There were 26,000 more babies conceived in 1996 in England and Wales than there had been one year previously. Since so many of the extra pregnancies were unplanned, that led to a lot more termination, too: 13,600 more abortions than in 1995.

This might seem a small price to pay to avoid a blood clot careening through your circulatory system, wreaking potentially lethal havoc. Think about all the women who were spared from death by embolism by the CSM’s warning!

But how many women, exactly, is that? We can’t know for sure. But one scientist, a supporter of the CSM decision to issue the warning, said the total number of embolism deaths prevented was “ possibly one.” The added risk posed by third-generation birth control pills, while significant in Fisher’s statistical sense, was not so significant in the sense of public health.

The way the story was framed only magnified the confusion. The CSM reported a risk ratio: third-generation pills doubled women’s risk of thrombosis. That sounds pretty bad, until you remember that thrombosis is really, really rare. Among women of childbearing age using first- and second-generation oral contraceptives, 1 in 7,000 could expect to suffer a thrombosis; users of the new pill indeed had twice as much risk, 2 in 7,000. But that’s still a very small risk, because of this certified math fact: twice a tiny number is a tiny number. How good or bad it is to double something depends on how big that something is! Playing ZYMURGY on a double word score on the Scrabble board is a triumph; hitting the same square with NOSE is a waste of a move.

Risk ratios are much easier for the brain to grasp than tiny splinters of probability like 1 in 7,000. But risk ratios applied to small probabilities can easily mislead you. A study by sociologists at CUNY found that infants cared for at in-home day cares or by nannies had a fatality rate seven times that of kids in day-care centers. But before you fire your au pair, consider for a minute that American infants hardly ever die these days, and when they do it’s almost never because a caregiver shook them to death. The annual rate of fatal accidents in home-based care was 1.6 per 100,000 babies: a lot higher, indeed, than the rate of 0.23 per 100,000 in day-care centers.* Both risks are small in absolute terms, though neither is zero. In the CUNY study, only a dozen or so babies a year died in accidents in family day cares, a tiny fraction of the 1,110 U.S. infants who died in accidents overall in 2010 (mostly by strangulation in bedclothes) or the 2,063 who died of sudden infant death syndrome. All things being equal, the results of the CUNY study provide a reason to prefer a day-care center to care in a family home; but all other things are usually not equal, and some inequalities matter more than others. What if the scrubbed and city-certified day-care center is twice as far from your house as the slightly questionable family-run in-home day care? Car accidents killed 79 infants in the U.S. in 2010. Spending 20% more time on the road would change exposure to that risk, but comparing it with the day-care difference requires comparable exposure data; the national death count alone does not establish which option is safer.

A significance test is a scientific instrument, and like any other instrument, it has a certain degree of precision. If you make the test more sensitive—by increasing the size of the studied population, for example—you enable yourself to see ever-smaller effects. That’s the power of the method, but also its danger. The truth is, the null hypothesis, if we take it literally, is probably just about always false. When you drop a powerful drug into a patient’s bloodstream, it’s hard to believe the intervention has exactly zero effect on the probability that the patient will develop esophageal cancer, or thrombosis, or bad breath. Every part of the body speaks to every other, in a complex feedback loop of influence and control. A sufficiently large study may detect an effect too small to matter for the decision at hand. Whether an observed association is causal also depends on the study’s design and assumptions. Just because we can detect them doesn’t always mean they matter.

If only we could go back in time to the dawn of statistical nomenclature and declare that a result passing Fisher’s test with a p-value of less than 0.05 was “statistically noticeable” or “statistically detectable” instead of “statistically significant”! That would be truer to the meaning of the method, which assesses the data against a null model but does not by itself establish an effect’s size, importance, or cause. But it’s too late for that. We have the language we have.*