Confusion of the Inverse (Prosecutor's Fallacy)

Confusion of the Inverse

Imagine a hypothetical Facebook algorithm that flags users as possible terrorism risks. It need not predict an individual crime. Think of something much more modest: a list of a hundred thousand users about whom Facebook can say, with some degree of confidence, “People drawn from this group are about twice as likely as the typical Facebook user to be terrorists or terrorism supporters.”

What would you do if you found out a guy on your block was on that list? Would you call the FBI?

Before you take that step, draw another box.

For this hypothetical example, suppose the box contains 200 million Facebook users in the United States. The line between the upper and lower halves separates future terrorists, on the top, from the innocent below. Any terrorist cells in the United States are surely pretty small—let’s say, to be as paranoid as possible, that there are ten thousand people who the feds really ought to have their eye on. That’s one in twenty thousand of the total user base.

The division between left and right is the one Facebook makes; on the left-hand side are the 100,000 people Facebook reckons as having an elevated chance of terrorist involvement. We’ll take Facebook at their word, that their algorithm is so good that the people who bear its mark are fully twice as likely as the average user to be terrorists. So among this group, 1 in 10,000, or 10 people, will turn out to be terrorists, while 99,990 will not.

If 10 out of the 10,000 future terrorists are in the upper left, that leaves 9,990 for the upper right. By the same reasoning: there are 199,990,000 nonoffenders in Facebook’s user base, 99,990 of whom were flagged by the algorithm and sit in the lower left box; that leaves 199,890,010 people in the lower right. If you add up all four quadrants, you get 200,000,000—that is, everybody.

Somewhere in the four-part box is your neighbor down the block.

But where? What you know is that he’s in the left half of the box, because Facebook has identified him as a person of interest.

And the thing to notice is that almost nobody in the left half of the box is a terrorist. In fact, there’s a 99.99% chance that your neighbor is innocent.

Being on the Facebook list doubles a person’s chance of being a terrorist, which sounds terrible. But that chance starts out very small, so when you double it, it’s still small.

But there’s another way to look at it, which highlights even more clearly just how confusing and treacherous reasoning about uncertainty can be. Ask yourself this—if a person is in fact not a future terrorist, what’s the chance that they’ll show up, unjustly, on Facebook’s list?

In the box, that means: if you’re in the bottom row, what’s the chance that you’re on the left-hand side?

That’s easy enough to compute; there are 199,990,000 people in the bottom half of the box, and of those, a mere 99,990 are on the left-hand side. So the chance that an innocent person will be marked as a potential terrorist by Facebook’s algorithm is

99,990/199,990,000

or about 0.05%.

That’s right—an innocent person has only a 1 in 2,000 chance of being wrongly identified as a terrorist by Facebook!

Now how do you feel about your neighbor?

The reasoning that governs p-values gives us clear guidance. The null hypothesis is that your neighbor is not a terrorist. Under that hypothesis—that is, presuming his innocence—the chance of him showing up on the Facebook red list is a mere 0.05%, well below the 1-in-20 threshold of statistical significance. Under that significance-testing rule, you might reject the null hypothesis. Treating that rejection as a declaration that your neighbor is a terrorist is precisely the mistake.

Except there’s a 99.99% chance he’s not a terrorist.

On the one hand, there’s hardly any chance that an innocent person will be flagged by the algorithm. At the same time, the people the algorithm points to are almost all innocent. It seems like a paradox, but it’s not. It’s just how things are. And if you take a deep breath and keep your eye on the box, you can’t go wrong.

Here’s the crux. There are really two questions you can ask. They sound kind of the same, but they’re not.

Question 1: What’s the chance that a person gets put on Facebook’s list, given that they’re not a terrorist?

Question 2: What’s the chance that a person’s not a terrorist, given that they’re on Facebook’s list?

One way you can tell these two questions are different is that they have different answers. Really different answers. We’ve already seen that the answer to the first question is about 1 in 2,000, while the answer to the second is 99.99%. And it’s the answer to the second question that you really want.

The quantities these questions contemplate are called conditional probabilities; “the probability that X is the case, given that Y is.” And what we’re wrestling with here is that the probability of X, given Y, is not the same as the probability of Y, given X.

If that sounds familiar, it should; it is the error of treating an unlikely observation under a hypothesis as a direct measure of that hypothesis’s plausibility. A common shorthand for the p-value is

“The chance that the observed experimental result would occur, given that the null hypothesis is correct.”

More precisely, a p-value includes results at least as incompatible with the null hypothesis as the observed result, under a specified testing procedure. It still conditions on the null hypothesis.

But what we want to know is the other conditional probability:

“The chance that the null hypothesis is correct, given that we observed a certain experimental result.”

The danger arises precisely when we confuse the second quantity for the first. And this confusion is everywhere, not just in scientific studies. When the district attorney leans into the jury box and announces, “There is only a one in five million, I repeat, a ONE IN FIVE MILLLLLLLION CHANCE that an INNOCENT MAN would match the DNA sample found at the scene,” he is answering question 1: How likely would an innocent person be to look guilty? But the jury’s job is to answer question 2: How likely is this guilty-looking defendant to be innocent? That’s a question the DA can’t help them with.*

—

The example of Facebook and the terrorists makes it clear why you should worry about bad algorithms as much as good ones. Maybe more. It’s creepy and bad when you’re pregnant and Target knows you’re pregnant. But it’s even creepier and worse if you’re not a terrorist and Facebook thinks you are.

You might well think that Facebook would never cook up a list of potential terrorists (or tax cheats, or pedophiles) or make the list public if they did. Why would they? Where’s the money in it? Maybe that’s right. But the NSA collects data on people in America, too, whether they’re on Facebook or not. Unless you think they’re recording the metadata of all our phone calls just so they can give cell phone companies good advice about where to build more signal towers, there’s something like the red list going on. Big Data isn’t magic, and it doesn’t tell the feds who’s a terrorist and who’s not. But it doesn’t have to be magic to generate long lists of people who are in some ways red-flagged, elevated-risk, “people of interest.” Most of the people on those lists will have nothing to do with terrorism. How confident are you that you’re not one of them?