October 19, 2012
In National Polling, It’s Gallup vs. the Rest
By NATE SILVER
The Gallup national tracking poll now shows a very strong lead for Mitt Romney. As of Friday, he was ahead by six points among likely voters, having led by seven points on Thursday.
However, the poll’s results are deeply inconsistent with the results that other polling firms are showing in the presidential race, and the Gallup poll has a history of performing very poorly when that is the case.
Other national polls now show a very slight lead for President Obama on average, while state polls continue to indicate a narrow advantage for the president in tipping-point states like Ohio. The FiveThirtyEight forecast has Mr. Obama as a modest favorite in the election largely on the basis of the state polls.
The Gallup poll is accounted for in the forecast model, along with all other state and national surveys.
There are two major pieces of information that we’re looking to extract from each poll. One is simply the raw number — who is ahead or behind? The other is the trend it shows in the race — which candidate is gaining or losing ground?
Different types of polls are relatively more or less useful for these purposes. Because national tracking polls like Gallup are published every day, they are useful for the trend part of the calculation.
There are six national tracking polls published most days. The others are from Rasmussen Reports, Ipsos, the RAND Corporation, Investors’ Business Daily and United Press International. (A seventh daily tracking poll, from Public Policy Polling, made its debut on Thursday.)
But of the daily tracking polls, the Gallup survey receives the largest weight in the model’s trendline calculation. It uses a larger sample size than most other polls, and its methodology includes calls to cellphone voters.
On the other hand, our pollster ratings are also based in part on past accuracy, and Gallup’s performance is middling in that department.
The Gallup poll seems to have an outsize influence on the subjective perception of where the presidential race stands, however — especially when the poll seems to diverge from the consensus.
This simply isn’t rational, in my view. Usually, when a poll is an outlier relative to the consensus, its results turn out badly.
You do not need to look any further than Gallup’s track record over the past few election cycles to see this.
In 2008, the Gallup poll put Mr. Obama 11 points ahead of John McCain on the eve of that November’s election. The average of the 15 or so national polls released just before the election put Mr. Obama up by about seven points.
The average did a good job; Mr. Obama won the popular vote by seven points. The Gallup poll had a four-point miss, however.
In 2010, Gallup put Republicans ahead by 15 points on the national Congressional ballot, higher than other polling firms, which put Republicans an average of 8 or 9 points ahead. In fact, Republicans won the popular vote for the House of Representatives by about seven percentage points — fairly close to the average of polls, but representing another big miss for Gallup.
The Gallup poll also has often found implausibly large swings within a race. In 2000, for example, Gallup had George W. Bush 16 points ahead of Al Gore among likely voters in polling it conducted in early August. By Sept. 20, about six weeks later, the firm had Mr. Gore up by 10 points instead: a 26-point swing over the course of a month and a half. No other polling firm showed a swing remotely that large.
Then in October 2000, Gallup showed a 14-point swing toward Mr. Bush over a few days, and had him ahead by 13 points on Oct. 27 — just 10 days before an election that ended in a virtual tie.
After the Republican convention in 2008, Gallup had Mr. McCain leading Mr. Obama by as many as 10 points among likely voters. Although some other polls also had Mr. McCain pulling ahead in the race, no other polling firm ever gave him more than a four-point lead.
It’s not clear what causes such large swings, although Gallup’s likely voter model may have something to do with it.
Even their registered voter numbers can be volatile, however. In early September of this year, after the Democratic conventions, they had Mr. Obama’s lead among registered voters going from seven points to zero points over the course of a week — and then reverting to six points just as quickly. Most other polling firms showed a roughly steady race during this time period.
Because Gallup’s polls usually take large sample sizes, statistical variance alone probably cannot account for these sorts of shifts. It seems to be an endemic issue with their methodology.
To be clear, I would not recommend disregarding the Gallup poll. You should consider it — but in context.
The context is that its most recent results differ substantially from the dozens of other state and national polls about the campaign. It’s much more likely that Gallup is wrong and everyone else is right than the other way around.