Blind Data Analysis

Blind Data Analysis

A hotly debated question in twentieth-century physics was whether the expansion of our universe is speeding up or slowing down. That matters in part because it tells us what the far future will look like: If expansion is speeding up, then all the matter in existence will keep getting farther and farther apart for all eternity. If expansion is slowing down, then everything will eventually collapse into a single point, like the big bang in reverse (the “big crunch”).

In the 1990s, physicist Saul Perlmutter ran the Supernova Cosmology Project, a team of researchers who investigated the universe’s changing speed of expansion by measuring the light given off by supernovae, or exploding stars. Personally, Perlmutter suspected the answer was going to turn out to be “expansion is speeding up.” But he was worried about the potential for motivated reasoning tainting the research process. He knew that even the most well-intentioned scientists can fool themselves and end up finding what they hope or expect to find in their data.

So Perlmutter opted for a method called blind data analysis. He used a computer program to shift all the supernova data by a random amount, which remained hidden from the researchers as they did their analyses. Because they couldn’t see the unshifted result, they had less opportunity to tune their analysis toward the answer they wanted. Only once all the analyses were finalized did the team get to see what the results looked like with the real data—and indeed, the “speeding up” theory was confirmed. Perlmutter would go on to win the Nobel Prize for the finding in 2015. Blind data analysis is “a lot more work in some sense, but I think it leaves you feeling much safer as you do your analysis,” he told a journalist.

It’s probably not every day that you find yourself testing a Nobel Prize–worthy theory about the nature of reality, but the same principle applies to more ordinary situations as well. Do you try to avoid biasing the information you get? For example, when you ask your friend to weigh in on a fight you had with your partner, do you describe the disagreement without revealing which side you were on, so as to avoid influencing your friend’s answer? When you launch a new project at work, do you decide ahead of time what will count as a success and what will count as a failure, so you’re not tempted to move the goalposts later?