The Haruspicy Parable

THE INTERNATIONAL JOURNAL OF HARUSPICY

Here’s a parable I learned from the statistician Cosma Shalizi.

Imagine yourself a haruspex; that is, your profession is to make predictions about future events by sacrificing sheep and then examining the features of their entrails, especially their livers. You do not, of course, consider your predictions to be reliable merely because you follow the practices commanded by the Etruscan deities. That would be ridiculous. You require evidence. And so you and your colleagues submit all your work to the peer-reviewed International Journal of Haruspicy, which demands without exception that all published results clear the bar of statistical significance.

Haruspicy, especially rigorous evidence-based haruspicy, is not an easy gig. For one thing, you spend a lot of your time spattered with blood and bile. For another, a lot of your experiments don’t work. You try to use sheep guts to predict the price of Apple stock, and you fail; you try to model Democratic vote share among Hispanics, and you fail; you try to estimate global oil supply, and you fail again. The gods are very picky and it’s not always clear precisely which arrangement of the internal organs and which precise incantations will reliably unlock the future. Sometimes different haruspices run the same experiment and it works for one but not the other—who knows why? It’s frustrating. Some days you feel like chucking it all and going to law school.

But it’s all worth it for those moments of discovery, where everything works, and you find that the texture and protrusions of the liver really do predict the severity of the following year’s flu season, and, with a silent thank-you to the gods, you publish.

You might find this happens about one time in twenty.

That’s what I’d expect, anyway. Because I, unlike you, don’t believe in haruspicy. I think the sheep’s guts don’t know anything about the flu data, and when they match up, it’s just luck. In other words, in every matter concerning divination from entrails, I’m a proponent of the null hypothesis. So in my world, it’s pretty unlikely that any given haruspectic experiment will succeed.

How unlikely? The standard threshold for statistical significance, and thus for publication in IJoH, is fixed by convention to be a p-value of .05, or 1 in 20. For a valid test at this threshold, if the null hypothesis is true, the chance of a statistically significant result is at most 1 in 20. If the null hypothesis is always true—that is, if haruspicy is undiluted hocus-pocus—we would expect about 1 in 20 experiments to be publishable under this rule.

And yet there are hundreds of haruspices, and thousands of ripped-open sheep, and even one in twenty divinations provides plenty of material to fill each issue of the journal with novel results, demonstrating the efficacy of the methods and the wisdom of the gods. A protocol that worked in one case and gets published usually fails when another haruspex tries it; but experiments without statistically significant results don’t get published, so no one ever finds out about the failure to replicate. And even if word starts getting around, there are always small differences the experts can point to that explain why the follow-up study didn’t succeed; after all, we know the protocol works, because we tested it and it had a statistically significant effect!

Modern medicine and social science are not haruspicy. But dissident scientists have been pounding out an uncomfortable message: there’s probably a lot more entrail reading in the sciences than we’d like to admit.

The loudest drummer is John Ioannidis, a Greek high school math star turned biomedical researcher whose 2005 paper “Why Most Published Research Findings Are False” touched off a fierce bout of self-criticism (and a second wave of self-defense) in the clinical sciences. Some papers plead for attention with a title more dramatic than the claims made in the body, but not this one. Ioannidis takes seriously the idea that entire specialties of medical research are “null fields,” like haruspicy, in which there are simply no actual effects to be found. “It can be proven,” he writes, “that most claimed research findings are false.”

“Proven” is a little more than this mathematician is willing to swallow, but Ioannidis certainly makes a strong case that his radical claim is not implausible. The story goes like this. In medicine, most interventions we try won’t work and most associations we test for are going to be absent. Think about tests of genetic association with diseases: there are lots of genes on the genome, and most of them don’t give you cancer or depression or make you fat or have any recognizable direct effect at all. Ioannidis asks us to consider the case of genetic influence on schizophrenia. Such an influence is almost certain, given what we know about the heritability of the disorder. But where is it on the genome? Researchers might cast their net wide—it’s the Big Data era, after all—looking at a hundred thousand genes (more precisely: genetic polymorphisms) to see which ones are associated with schizophrenia. In this hypothetical case, Ioannidis suggests that around ten of these actually have some clinically relevant effect.

And the other 99,990? They’ve got nothing to do with schizophrenia. But with unadjusted tests at this threshold, about 1 in 20 of them, or five thousand, would be expected to pass the test of statistical significance. In other words, among the “OMG I found the schizophrenia gene” results that might get published, there are five hundred times as many bogus ones as real ones.

And that’s assuming that all the genes that really do have an effect on schizophrenia pass the test! A study can fail to detect a real effect if it lacks the statistical power to find it. If the studies are underpowered, the genes that truly do make a difference might pass the significance test only half the time; but that means that of the associations declared statistically significant, only five would be genuine, as against the five thousand pretenders that passed the test by luck alone.

A good way to keep track of the relevant quantities is by drawing circles in a box:

The size of each circle represents the number of genes in each category. On the left half of the box we have the negatives, the genes that don’t pass the significance test, and on the right half we have the positives. The two top squares represent the tiny population of genes that actually do affect schizophrenia, so the genes in the top right are the true positives (genes that matter, and the test says they matter) while the top left represents the false negatives (genes that matter, but the test says they don’t). In the bottom row, you have the genes that don’t matter; the true negatives are the big circle on the bottom left, the false positives the circle on the bottom right.

You can see from the picture that the significance test isn’t the problem. It’s doing exactly the job it’s built to do. The genes that don’t affect schizophrenia very rarely pass the test, while the genes we’re really interested in pass half the time. But the nonactive genes are so massively preponderant that the circle of false positives, while small relative to the true negatives, is much larger than the circle of true positives.