Dr. Pete on Public Opinion Polling
1/1/0001
Dr. Pete
Dear Dr. Pete:
How come the government can get a 70% response rate to its surveys and I struggle to get 10%? I'm not just talking about the census here-they seem to do it all the time.
-William Weaver
Dear Bill:
Several things here...
- It matters more.
- They care more.
- They work harder at it.
- They have (your) money to throw at the problem.
Why do they think it matters more? What is it about the response rate that is so important? Theory tells us that there is a non-response bias. Put bluntly, you have no idea what the 90% who won't, or can't, take your survey think about the subject in hand. What if they think differently than the 10% who take the survey? If that is the case, your final report to the client could be very flawed. Does this happen in practice? Generally speaking, I have to say no. The bias of course is there. The majority do not take part in the research. But is it correlated to the subject at hand? Most times no, since for telephone, face-to-face and online research, the refusal takes place before any consideration of the subject matter takes place.
What do I mean by correlated? Think about it this way-if men and women thought exactly the same about any issue (yes, I know, but stay with me on this....) then it wouldn't matter how many women and how many men there were in your sample. The answer would still be right. The subject matter is not correlated to gender. Then why do we regularly quota control our samples on age and gender? Because we don't know what variables will correlate with the subject (i.e., we don't know what to control on) and we simply hope that age and gender will make it work.
This kind of "wing and a prayer" attitude to non-response doesn't wash with postal surveys. With postal surveys, there is often a clear correlation between non-response and the subject matter, materially affecting the survey outcome. Peter Ognibene, in a seminal paper from 1971, (and I think you will see that many papers that give their conclusions on page one and are therefore clearly visible on the free bits of Jstor will be described as seminal as time goes on) found under-reporting of readership of a newspaper by 27%. While it's unclear whether this is 27% or 27 percentage points, it's still a lot.
Notice how the researcher was able to identify that some bias was working on his data by reference to some known variable. Pretty sneaky, hey? Maybe we should all think of doing something similar. Line up the known data and then make the assumption that your concept test scores are also a-okay. Then, of course, if you find non-alignment in the known variable, you are going to have to deal with it, probably at your own expense. Maybe then, on second thought, we should not.
So how can we fix this scourge? Within pure random sampling methods sometimes the non-response bias will be visible in terms of demographics and weighting can be used to correct it. Of course, this affects the effective sample size so you are going to need to do that many more interviews. Other times the bias will be less visible and only manifest itself in the data not lining up as it should. In this instance it may be necessary to undertake a further survey of non-responders, possibly by a different methodology, to try to get estimates for this group.
In the Ognibene Case (as I like to call it), a sample of non-responders was interviewed by phone. The data they gave brought the readership data back in line. Actually if you read the paper, you might notice that the members of the survey population were already responders to an original telephone survey in which these questions had been asked. I wonder why Mr. Ognibene didn't simply refer to the original answers to discover the nature of the non-response bias. Hey ho, it was a long time ago and I hope Peter won't mind me mentioning it now.
Final thought for those of us who don't work with taxpayers' hard-earned cash. Ever heard this from a client? "I'd like you to maximize your response rate and please deliver the survey results within 3 days." Well, something's gotta give here. Some non-response is, of course, non-availability over the fieldwork period. If you make the fieldwork period longer and remind the respondent, then you are going to increase your response rate. However, miss the reporting deadline and you might not get engaged again. Guess which one I'm going to do.... "We did everything in our power to maximize response yada, yada, yada and the response rate was a very respectable 10%."
Good hunting,
-Dr. Pete
Dear Dr. Pete:
Still reckon opinion polls are accurate?
-A S Ceptic
Dear A S:
Yep, still do. Remember back in August '08 we looked at the current state of polls in the McCain (remember him) Obama race? The last Gallup Daily at that time had Obama getting around 55% share and winning by 9%. He got 52.9% and won by 7.2%.
I stand by my oft-stated views that if I could predict the "next big thing" as accurately as this, I'd be a very rich man.
-Dr. Pete
