Sunday, Nov. 7, 2010 | 2 a.m.
- Election polls under scrutiny after missing mark (11-4-2010)
- Poll shows Angle, Reid close in Nevada Senate race (10-29-2010)
- Jon Ralston: The consequences of flawed public polling (10-8-2010)
- Harry Reid inching ahead of Sharron Angle, new poll finds (9-25-2010)
- Polls: Harry Reid grabs lead over Sharron Angle (7-16-2010)
- Poll: Either GOP challenger could topple Harry Reid (8-23-2009)
- Margin of error is polls’ fine print (10-14-2008)
All the late public polling in the race between Sen. Harry Reid and his Republican challenger Sharron Angle turned out to be wrong, including the Las Vegas Review-Journal’s final Mason-Dixon poll that had Angle up 49-45. Reid won by 5.
But Reid pollster Mark Mellman nailed it. Just as he did in the California and West Virginia Senate races.
How did he do it? And what was wrong with the public polls?
Mellman was kind of enough to take some time last week to walk me through it, and here are the key points:
• Pollsters too often subscribe to the “likely voter” fallacy. They seek out likely voters by asking them questions about their past behavior, which should give some indication of whether they’ll actually vote.
But Mellman looks for the likely electorate. The distinction is important. As Mellman notes, there are “unlikely voters” who actually wind up voting. And “likely voters” who wind up not voting. Every election there is some percentage of both, and Mellman has clearly mastered the art and science of modeling the likely electorate. It seems that in the Reid race, unlikely Reid voters — Hispanics come to mind — actually voted, with the help of the Reid turnout operation and Angle’s immigration ads that were deeply offensive to these voters.
• The cell phone problem. By now, this one is fairly well known, but as Mellman said, “It is a real problem.” To begin with, federal law prohibits automated calls to cell phones. So any poll conducted that way can’t call any cell phones. “There’s a real difference between people we reach on land lines and people on cell phones. If you’re only calling land lines, you get a distorted picture.”
A Pew Research Center analysis showed that a quarter of all Americans can be reached only by cell phone. They tend to be younger, a demographic dominated by Democrats. Pew estimated the land-line bias gives Republicans 4 to 6 percentage points in polls.
• Hard-to-reach voters. People don’t pick up their phones. A truly random sample requires the pollster to be persistent and try and try again with the same voter. Otherwise, you don’t get a truly random sample; you get a random sample of easy-to-reach voters.
Again, this has a distorting effect. Mellman’s group will try people six or seven times. Many pollsters are not as persistent because it’s cheaper to just move along.
“In Nevada these factors all mattered and all mattered meaningfully,” Mellman said.
Mellman didn’t address Hispanic voters, but polling experts including Nate Silver have speculated they may have been undercounted in polls.
(Sun columnist Jon Ralston was the earliest and most vocal critic of public polling in the race.)
The Reid camp said all along that it was confident because of its internal polls. Reporters were rightly skeptical, especially given that the campaign wasn’t sharing the data.
The key, though, is that the Reid campaign wasn’t using its polls as part of a press strategy, trying to goose up supporters. They wanted good information from which to craft a strategy.
“There’s no sense paying for bad information,” Mellman said. “Our job starts at getting it right. Then we use and interpret the data to build a strategy. But getting it right is the foundation.”
Mellman addressed the potentially damaging effect of bad public polling. “The problem is that these things can have an impact.” A bad poll can hurt a campaign, creating a narrative of defeat in the media, deflating donors and volunteers.
“So the media has a special responsibility to get it right. If you’re not going to get it right, then don’t bother.”
A novel concept in some quarters.