LocalCommentary.com
David Rodeback's Blog

Thursday, September 16, 2004

A Half-Baked Story about a Half-Baked Poll

Bloomberg, a large investment information company, published a story yesterday at Bloomberg.com with the headline "Bush Ratings Drop among Uncommitted Voters, Annenberg Poll Says." No doubt the media in general will pick up the same story soon. But there's a gaping hole in the methodology, and therefore in the conclusion.

It seems that President Bush's approval rating among undecided voters dropped from 56 percent before the Republican National Convention two weeks ago, to 44 percent after the convention. This is an overall figure, but his rating on several specific key issues fell, too. At first blush, it sounds as if the convention gave the President a large negative bounce among undecided voters. This contradicts about a dozen other polls, but if you look carefully, you may see why.

I followed the story to the Annenberg Public Policy Center at the University of Pennsylvania, which conducted the poll and released the results. Their long and detailed press release calls the voters in question "persuadable voters"; it's a friendlier term than "undecided" in the current climate, when one would think only the hopelessly indecisive or the criminally uninformed could still be undecided. For this poll's purposes, persuadable voters are "respondents who said they were undecided among Bush, John Kerry or Ralph Nader for president or who said they had a preference but there was a 'good chance' they would change their minds."

The Annenberg release ventures several possible reasons for this shift, but they do not account for the most obvious one. Suppose that a certain percentage of the undecided voters are more susceptible than others to being persuaded to the President's camp. That's almost inevitable. Now suppose that they were paying attention to the convention and found themselves persuaded. Here's the point: When they made up their minds, they ceased to be undecided, and therefore would not have been counted as undecided for purposes of the second poll.

It stands to reason that if the most pro-Bush undecided voters are removed from the pool, and then the President's approval in that pool is measured again, it will have declined substantially - not because he fared poorly among them, but because he succeeded in swaying some of them, who are therefore no longer counted as undecided in the poll.

The Annenberg release includes the poll questions and results and a discussion of methodology. The best clue that what I have described actually happened is that President Bush's approval in the total sample (not just the undecided voters) increased, while Senator Kerry's decreased.

The obvious way to control for this problem is to poll the very same voters in the second poll as were polled in the first. Neither the Bloomberg article nor the Annenberg press release offers the slightest hint that this was done, or that this problem was even considered in the poll design.

In short, the statistics themselves appear to show exactly the opposite shift from what is being reported. Whether the analysis is accidentally or deliberately misleading - whether it is agenda-driven or simply clueless - doesn't matter much. Either way, it's unprofessional, and it preys upon people who lack the time, inclination, or skills to analyze what is reported in the media.

Ironically, the Annenberg release ends with this little plug: "Another major election project of the Annenberg Public Policy Center is FactCheck.org, a project that tries to hold politicians accountable by exposing false or misleading campaign statements. It is available online at www.FactCheck.Org."

Ahem. Is anyone else smiling just now?

Copyright 2004 by David Rodeback.
All rights reserved.
Permission is given to publish this article elsewhere on the Web or via e-mail only under these conditions: The article is published intact and unchanged, and is properly credited to the author; and a visible link is included to the article's page at LocalCommentary.com.