http://dyn.politico.com/printstory.cfm?uuid=9AB72D23-35E1-498E-A39B-9EA20E26E7BA Gallup seeks fixes after 2012 miss
By: Steven Shepard
May 16, 2014 06:06 PM EDT
ANAHEIM, Calif. — Eighteen months after the Gallup Organization incorrectly showed Mitt Romney leading President Barack Obama on the eve of the 2012 presidential election, the legendary but embattled polling firm unveiled additional research aimed at fine-tuning their methodology before the 2016 campaign.
Gallup conducted polls last year in New Jersey and Virginia in the run-up to gubernatorial elections in those states, testing different modes of surveying voters and alternative methods for accurately predicting which voters would actually cast ballots.
Overall, the research showed telephone polling was more accurate than Internet surveys the firm commissioned – suggesting that any switch towards the web, as some news organizations have done, was premature for their purposes.
“We simply haven’t found another way to estimate the national electorate,” said Gallup editor-in-chief Frank Newport in an interview after the research was presented.
Moreover, the review found that Gallup’s likely-voter screen – a battery of questions, including self-reported vote history and present-day intentions – produced more accurate results than other methods of filtering out voters who won’t end up casting ballots.
The results, the second phase of Gallup’s review of its 2012 Republican bias, were released for the first time here Friday morning at the American Association for Public Opinion Research’s annual conference. The polls in New Jersey and Virginia were conducted exclusively for the purpose of research and were not released to the public. Aiding Gallup in their analysis was University of Michigan professor Michael Traugott, a past president of the association and expert in survey research, and graduate students at Michigan and the University of Maryland.
Gallup’s 2012 final likely-voter estimate showed Romney leading Obama by one point, 49 percent to 48 percent. Among all registered voters, Obama led, 49 percent to 46 percent. That Gallup’s registered-voter results were closer to Obama’s eventual, four-point margin of victory means that their efforts to model the electorate achieved the opposite result. They excluded voters who said they weren’t giving much thought to the election, though some of these voters turned out on Election Day, mostly for Obama.
But Gallup’s research fell short of addressing these issues for some in attendance. ABC News’s Dan Merkle, who moderated the session at which three research papers were presented, noted that Gallup failed to experiment with tweaks to the “thought-given-to-the-election” question as part of the likely-voter model. And he pointed out that Gallup’s error in the New Jersey gubernatorial race – it showed GOP Gov. Chris Christie leading by a margin five points greater than his ultimate advantage – was equal to their five-point error in the 2012 presidential race. (Gallup’s poll in Virginia was closer, only slightly underestimating Democrat Terry McAuliffe’s winning margin, showing him ahead by one point instead of three.)
The New Jersey and Virginia phone surveys revisited some familiar themes for Gallup. Like in 2012, its likely-voter universe included a larger percentage of white voters than were measured by exit polls. Both polls also underestimated Democrats’ composition of the electorate, compared to the exit polls. Democrats held a 12-point edge over Republicans in the 2013 exit poll in New Jersey, but Gallup’s pre-election survey showed Democrats with just a two-point edge among likely voters.
In June 2013, when Gallup presented the first phase of its review, Newport told MSNBC’s “Morning Joe” that the likely-voter model “probably needed a total overhaul.” But the experiments conducted in this phase of research involved more subtle changes.
Gallup’s likely-voter model is largely based on self-reporting. Respondents tell interviewers whether they are registered to vote, whether they have voted in past elections, whether they intend to vote this year and how interested they are in the election. To that end, Gallup and researchers at Michigan designed two experiments to reduce the degree of over-reporting – people saying they voted in the past when they haven’t, for example.
By asking questions aimed at source monitoring – prompting New Jersey and Virginia voters to think about specific details about whether they had voted in the last gubernatorial election in 2009 – researchers found that reduced the number of voters who said they had cast ballots by between five and seven percentage points. Similarly, prompting voters to think about things that might prevent them from voting, such as an illness on Election Day, reduced the percentage who said they intended to vote immediately prior to the 2013 election.
In addition to difficulties identifying likely voters, Gallup initially pointed to oversampling of some regions, the way it measured race and ethnicity and the exclusion of unlisted phone numbers as reasons for its 2012 inaccuracies.
Some of these issues were addressed immediately after last year’s report, but other questions were left unexplored during this second round of research.
The Huffington Post’s Mark Blumenthal asked why Gallup didn’t experiment with different methods of phone sampling. The telephone research only involved surveys conducted by randomly dialing phone numbers, instead of using voter lists to reach those who have turned out in the past. By using voter lists, Gallup could have gone back to verify that the voters they contacted ended up casting ballots after the election – and those they excluded did not.
Newport, Gallup’s editor-in-chief, demurred when asked if the firm had specific plans to commission further research in preparation for 2016.
“We’re always evolving and making decisions on what we want to do next,” he said.
Traugott said he’d like to continue working with Gallup to research more effective pre-election polling methods. “There’s more work to be done,” he said.
But Gallup hasn’t yet made a financial commitment to continue commissioning surveys purely for research purposes.
“Quite honestly, we’re using Gallup’s money to do this,” said Traugott.
