Probabilistic Forecasting and Uncertainty (Nate Silver)

A MAN WHO SWINGS FROM POLL TO POLL

The paladin of principled uncertainty in our time is Nate Silver, the online-poker-player-turned-baseball-statistics-maven-turned-political-analyst whose New York Times columns about the 2012 presidential election drew more public attention to the methods of probability theory than they have ever before enjoyed. I think of Silver as a kind of Kurt Cobain of probability. Both were devoted to cultural practices that had previously been confined to a small, inward-looking cadre of true believers (for Silver, quantitative forecasting of sports and politics; for Cobain, punk rock). And both proved that if you carried their practice out in public, with an approachable style but without compromising the source material, you could make it massively popular.

What made Silver so good? In large part, it’s that he was willing to talk about uncertainty, willing to treat uncertainty not as a sign of weakness but as a real thing in the world, a thing that can be studied with scientific rigor and employed to good effect. If it’s September 2012 and you ask a bunch of political pundits, “Who’s going to be elected president in November?” a bunch of them are going to say, “Obama is,” and a somewhat smaller bunch are going to say, “Romney is,” and the point is that all of those people are wrong, because the right answer is the kind of answer that Silver, almost alone in the broad-reach media, was willing to give: “Either one might win, but Obama is substantially more likely to win.”

Traditional political types greeted this response with disrespect. They wanted an answer. They didn’t understand that Silver was giving them one.

Josh Jordan, in the National Review, wrote: “On September 30, leading into the debates, Silver gave Obama an 85 percent chance and predicted an Electoral College count of 320−218. Today, the margins have narrowed—but Silver still gives Obama a 67 percent chance and an Electoral College lead of 288−250, which has led many to wonder if he has observed the same movement to Romney over the past three weeks as everyone else has.”

Had Silver noticed the movement to Romney? Clearly, yes. He gave Romney a 15% chance of winning at the end of September, and a 33% chance on October 22—more than twice as much. But Jordan didn’t notice that Silver had noticed, because Silver was still estimating that Obama had a better chance of winning than Romney did. For traditional political reporters like Jordan, that meant his answer hadn’t changed.

Or take Dylan Byers in Politico: “So should Mitt Romney win on Nov. 6, it’s difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance of winning (way back on June 2) and—one week from the election—gives him a one-in-four chance, even as the polls have him almost neck-and-neck with the incumbent. . . . For all the confidence Silver puts in his predictions, he often gives the impression of hedging.”

If you care at all about math, this is the kind of thing that makes you want to stab yourself in the hand with a fork. What Silver offers isn’t hedging; it’s honesty. When the weather report says there’s a 40% chance of rain, and it rains, do you lose faith in its predictions? No—you recognize that the weather is inherently uncertain, and that a definitive statement of whether it will or won’t rain tomorrow is usually the wrong thing to offer.*

Of course, Obama did win in the end, and with a comfortable margin, leaving Silver’s critics looking a little foolish.

The irony is that if the critics had wanted to catch Silver in a mistaken prediction, they missed a great chance. They could have asked him, “How many states are you going to get wrong?” As far as I know, nobody ever asked Silver this question, but it’s easy to figure out how he would have answered it. On October 26, Silver estimated that Obama had a 69% chance of winning New Hampshire. If you forced him to predict then and there, he’d call it for Obama. So you could say that Silver estimated his chance of being wrong about New Hampshire to be 0.31. Put in other words, the expected number of wrong answers he would give about New Hampshire was 0.31. Remember—the expected value isn’t the value you expect, but rather a probabilistic compromise among the possible outcomes—in this case, he’s either going to give zero wrong answers about New Hampshire (an outcome with probability 0.69) or one (an outcome with probability 0.31), which gives an expected value of

(0.69) × 0 + (0.31) × 1 = 0.31

via the method we set up in chapter 11.

Silver was more certain about North Carolina, giving Obama only a 19% chance of winning. But that still means he estimated a 19% probability that his Romney call would end up wrong; that is, he gave himself another 0.19 expected wrong answers. Here’s a list of the states Silver considered potentially competitive on October 26:

State	Obama win probability	Expected wrong answers
OR	99%	.01
NM	97%	.03
MN	97%	.03
MI	98%	.02
PA	94%	.06
WI	86%	.14
NV	78%	.22
OH	75%	.25
NH	69%	.31
IA	68%	.32
CO	57%	.43
VA	54%	.46
FL	35%	.35
NC	19%	.19
MO	2%	.02
AZ	3%	.03
MT	2%	.02

Since expected value is additive, the expected number of competitive states he’d pick wrong is the sum of these contributions, which comes to 2.83. In other words, he’d probably have said, if anyone had asked him, “On average I’m likely to get about three states wrong.”

Actually, he got all fifty right.*

—

Even the most seasoned political pundit might have trouble pulling off an attack on Silver for being more accurate than he said he would be. The twistiness this incites in the mind is healthy; follow it! When you reason correctly, as Silver does, you find that you always think you’re right, but you don’t think you’re always right. As the philosopher W. V. O. Quine put it, “To believe something is to believe that it is true; therefore a reasonable person believes each of his beliefs to be true; yet experience has taught him to expect that some of his beliefs, he knows not which, will turn out to be false. A reasonable person believes, in short, that each of his beliefs is true and that some of them are false.”

Formally, this is very similar to the apparent contradictions in American public opinion we unraveled in chapter 17. The American people think each government program is worthy of continued funding, but that doesn’t mean they think all government programs are worthy of continued funding.

Silver bypassed the sclerotic conventions of political reporting and told the public a truer story. Instead of saying who was going to win, or who had the “momentum,” he reported what he thought the chances were. Instead of saying how many electoral votes Obama was going to win, he presented a probability distribution: say, Obama had a 67% chance of getting the 270 electoral votes he needed for reelection, a 44% chance of breaking 300, a 21% chance of getting to 330, and so on. Silver was being uncertain, rigorously uncertain, in public, and the public ate it up. I wouldn’t have thought it was possible.

This is action, this not being sure!