Chris Correa reflects on whether we should listen to journalists at all when it comes to polling results, as NPR suggests that most journalist might be "innumerates:"
One of the rarely admitted secrets about journalists is that many of us are functional "innumerates" -- another way of saying "mathematically illiterate." Oh sure, we can add and subtract reasonably well. But with some exceptions, journalists generally don't know, understand or aren't interested in numbers. As for more complex subjects such as statistics and probability, well... many journalists would be hard pressed to tell the difference between "average" and "mean."
Chris says "mean" is the arithmetic mean, while "average" could be any measure of central tendency (mean, median, or mode). I confess I've never heard "average" used for anything other than the arithmetic mean, but then I'm around statisticians all day, and we're pretty precise. Has anyone else ever heard of using that to mean a general measure of central tendency?
Anyway, the rest of the NPR article is quite good, and supplies a few guidelines for interpreting poll results yourself:
One of the clearest explanations of what is trustworthy and what is less trustworthy about polling comes from professor Kathleen Woodruff Wickham of the University of Mississippi in Oxford.
Wickham says that voters need to ask the following questions whenever a poll is published:
Who sponsored the survey?
How many people were interviewed (sample size)?
Were the interviews done with a random or a self-selected group? This is important because of the increased use of Internet polling, where people may be more motivated to respond than if they were telephoned at random.
What is the wording of the question? Is the language emotional? Neutral?
What is the raw data? Do the math yourself, if you can.
To which I'd add:
1. Is the group representative of the population to which it should generalize? In plain English, that means if you want to know how the US college-age student population feels about something, you survey a large sample of US college students. Sound intuitive, but you'd be amazed how many researchers attempt to generalize to a population from which they have neglected to sample. Surveying a group of college students will not tell you who is going to be elected President this year, because (a) college students skew more left-wing than the rest of the population, and (b) 18-24-year-olds don't vote to the same extent as older adults.
2. How was the poll collected? Even if the sample is not self-selected, finding people through telephone vs. door-to-door vs. direct mailing vs. email is going to get you very different populations. Plus, if the topic of the poll is emotional (Lord knows the current election is), getting information over the phone vs. over the internet vs. face-to-face may produce different results.
3. How forthcoming are the pollsters with the raw data? Bottom line is, if it's not posted on the web, you should be able to email them or call them to get the information, and they should be happy to provide it. If they're not, be suspicious.
I took an entire course on survey sampling in college, which is a tad much for anyone who's not going to spend their life doing that. But I enjoyed it.
Update: Tall, Dark, and Mysterious has comments. I have to admit, blogs like his make me wish I was single (hee).
Posted by kswygert at October 11, 2004 09:40 PM