Skip to Content
5Nov 2012

In defense of Nate Silver: Pundits bare their misunderstanding.

Nate Silver

I feel sorry for Dylan Byers, a media blogger at Politico, and, from what I read, an entertaining and competent writer.

I'm embarrassed for David Brooks, a conservative columnist at The New York Times and a smart guy who writes about human nature in addition to politics.

As for Joe Scarborough, I'm always happy to catch a few minutes of his intelligent msnbc morning show, Morning Joe, when my kids aren't watching Phineas and Ferb. But I'm cringing at his remarks, too.

Each of these folks has plainly and publicly revealed a profound misunderstanding of the work of Nate Silver, the computer modeler and author of the FiveThirtyEight blog at The New York Times. And they are not alone. (At the end of this post, I'll link to a few commentators who got it right.)

Byers conveniently collected--and exemplified--the misunderstanding in his Oct. 29 column, which begins: "Nate Silver could be a one-term celebrity." 

First, here is Byers:

Prediction is the name of Silver's game, the basis for his celebrity. So should Mitt Romney win on Nov. 6, it's difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance of winning (way back on June 2) and — one week from the election — gives him a one-in-four chance, even as the polls have him almost neck-and-neck with the incumbent.

Byers interviewed Silver and reports his response. Silver warned Byers against confusing prediction with prophecy. He told Byers, "If the Giants lead the Redskins 24-21 in the fourth quarter, it's a close game that either team could win. But it's also not a "toss-up": The Giants are favored."

Here's my response to Byers: Let's compare Silver's work to a weather forecast. As of Nov. 4, Silver gives Obama an 86.3 percent chance of winning the election. If a meteorologist said there was an 86 percent chance of rain--and it didn't rain--Byers would presumably "not continue to put faith in the predictions" of the weather forecaster. But we know that's not right. Forecasts are generally correct--but not always. That does not make them worthless. When there is an 86 percent change of rain, most of us grab an umbrella. And we should.

Now here's Brooks, as reported in Byers's column:

 The pollsters tell us what's happening now. When they start projecting, they're getting into silly land.

And my response: Silver is not a pollster. He is not polling people. He is using data to make predictions. Researchers do this all the time, for such things as predicting animal cell behavior and predicting that Hurricane Sandy would ravage New York and New Jersey with unprecedented storm surges. 

And here's Morning Joe, again as quoted by Byers:

Nate Silver says this is a 73.6 percent chance that the president is going to win? Nobody in that campaign thinks they have a 73 percent chance — they think they have a 50.1 percent chance of winning. And you talk to the Romney people, it's the same thing," Scarborough said. "Both sides understand that it is close, and it could go either way. And anybody that thinks that this race is anything but a toss-up right now is such an ideologue, they should be kept away from typewriters, computers, laptops and microphones for the next 10 days, because they're jokes.

My response: Silver is not saying Obama will win by a margin of 70 or 80 percent. He's saying, as of the latest data, there is an 86.3 percent chance Obama will win the election. By all accounts--including Silver's--the election will be very close. Silver predicts that Obama will get 50.6 percent of the popular vote. Scarborough might not like that, but it fits with the consensus view that this will be a close election.

Even if Romney wins, Silver isn't wrong. Again, our weather example: When there is a 75% chance of rain (and a 25% chance of no rain) sometimes no rain is what we get. You can bet, however, that if Romney does win, these pundits will pounce on Silver, agreeing that he has proven himself to be a joke, or a one-term celebrity. And they will be wrong then, too. It's a prediction, not a prophecy.

One person who got it right--and who helped me understand the issues--was Zeynep Tufekci at Wired. Tufekci, an assistant professor at the University of North Carolina who works on complex systems wrote, in a splendid defense of Silver, that this "isn't wizardry," but "the sound science of complex systems." And here, she writes, is the point that the pundits have clearly missed: "Uncertainty is an integral part of it. But that uncertainty shouldn’t suggest that we don’t know anything, that we’re completely in the dark, that everything’s a toss-up."

And this is what she wrote that finally made Silver's work clear to me:

What his model says is that currently, given what we know, if we run a gabazillion modeled elections, Obama wins 80 percent of the time...Since we’ll only have one election on Nov. 6, it’s possible that Obama can lose. But Nate Silver’s (and others’) statistical models remain robust and worth keeping and expanding — regardless of the outcome this Tuesday.

For a more informed defense of Silver than mine, you can read Ezra Klein's Wonkblog at The Washington Post. Klein links to others--part of what he calls "the burgeoning world of election quants"--who are getting similar results using mathematical techniques. At the Nieman Journalism LabJonathan Stray has another intelligent commentary, in which he says that Silver's technique "has lessons for all journalists." (Thanks to Tabitha M. Powledge at On Science Blogs for these links.)

I'm sorry to report that Margaret Sullivan, the new public editor at The New York Times, also erred in describing Silver's work, writing that Silver "has been giving President Obama approximately a 75 percent chance of winning the election, at a time when various public opinion polls have been calling the race 'neck and neck.'" Silver does predict a neck-and-neck race, as I've noted above. The reason I'm sorry to report this is that Sullivan, still relatively new to the job, promises to be the best and hardest-working public editor the Times has yet had. I don't like to see her stumble.

I've tried to make this argument as simply and clearly as I can, but I'm taking a chance by criticizing others here. Statistics is a difficult science, and the inner workings of models like Silver's are far beyond my understanding. If I've made errors here, I hope they will be pointed out in the comments. And if anyone feels embarrassed for me, I'll take my lumps.

-Paul Raeburn

 

Comments

Regarding the use of quotes from other sources, we report here on science coverage. I quote from other stories and blog posts to show what I think was done well and what was done poorly.

But journalists and bloggers use quotes from secondary sources all the time. I've quoted various presidents of the United States on multiple occasions, and I've never spoken to any of them. The accepted practice is to make the source of the quotes clear if they do not come directly from the person quoted. 

 

You needed to check with David Brooks and Morning Joe to confirm that what was reported was what they actually said before you respond.  Original sources are required.  If you use second-hand sources -- which you did for those two -- you are on shaky ground journalistically.

It is frustrating indeed that so many people confuse the probability of an overall win for candidate with the probability that a single random person supports that candidate.

One way to explain the difference is to think of a future when the uncertainty is lower: On Wednesday, we will know the winner with near certainty (I profoundly hope!).   But the vote totals will still be nearly matched, near 50/50, even though the outcome has become 100/0 or 0/100.

Andrew Gelman is endorsing a commenter's suggestion that, instead of saying there's an 80% chance Obama will win, one should say "I predict Obama will win, but there's a 20% chance I'm wrong."

Boyce,

Your definition of a science writer is on target, but you're selling us short. I also had a thriving medical practice when I was in the newsroom. 

A good way to explain what Silver does is: Calculate the odds that a given candidate will win more than 50 percent of the votes cast.  (This definition ignores third party candidates.)

It is, indeed, sad to see that so many otherwise respectable people in journalism are so innumerate. This reinforces my definition of a science writer--the person in the newsroom to whom other journalists go when they need a percentage calculated.

--Boyce Rensberger

 

 

great post.    for  what it's worth, here's another story on the difficulty of sampling the electorate:

http://www.philly.com/philly/news/homepage/20121016_Voter_survey_disproportions_linked_to_difficult-to-reach_cellphone_users.html

 

I don't know how, but we should keep this conversation going until some of those who don't get it start seeing the meaning in the numbers. We can understand our world a lot better if we can sort the odds and the outcomes. Nice piece, Paul.

Login or register to post comments