It may not have been stormy, but it was dark last night when the Globe and Mail told a story on its Website about a poll of Canadians’ attitudes about health care – specifically, their supposed willingness to pay pots of money to private insurers for it.
“The survey shows strong support for user fees and having well-to-do Canadians pay more out-of-pocket to help attenuate the impact of caring for a growing population of seniors,” reported the Globe. No doubt this will be on Page 1 in the morning.
This must be true, readers surely concluded, because the Globe is Canada’s National Website and it says right there in grey and white that this poll is “considered accurate within 1.66 percentage points, 19 times out of 20.” Sure sounds scientific!
The poll was done by Ipsos-Reid (sounds reliable) for the Canadian Medical Association (if you can’t trust your doctor…), more evidence that it must be bang on. What’s more, it was immediately followed by a veritable flood of positive commentary from “ordinary Canadians” who, like this blogger, apparently have time on their hands and a keyboard in front of them during the long hours of darkness.
So if it just happened that the CMA’s results dovetailed precisely with the CMA’s current policy agenda, never mind. Coincidences happen…
However, here are some things to think about before you rush out and buy health insurance on the strength of this public opinion poll.
First, should we believe results collected with an on-line poll, as opposed to a poll conducted over respondents’ phone lines?
The trouble with polls collected on line is that the respondents select themselves. This means such polls are less likely to produce results that really reflect the views of the population being sampled.
Ipsos-Reid’s respondents come to the game with knowledge of polling, an inclination to take part, and quite possibly an agenda they want to push.
So, from the get-go, readers should be skeptical that this is truly a random sample. Readers should also ask: Why didn’t the CMA demand a more rigorous sampling method?
Second, while the sample of 3,483 respondents is a big one, the claim it can be considered accurate to a given percentage point a given number of times is highly controversial within professional polling circles.
The Canadian Marketing Research and Intelligence Association says pollsters shouldn’t publish a margin or error on Internet samples because on-line surveys are convenient samples, not random samples.
So why is Ipsos-Reid doing this? The simple explanation is because the statement lends credibility to the poll, something any pollster hopes to project, for obvious reasons.
Readers, however, should know this information – it was part of the Globe’s job to tell them about it. Perhaps the pressure of deadlines didn’t leave the reporter sufficient time to research the CMA’s press release properly.
Third, we need to know the order in which the questions were asked. It sounds very much as if the order of questions in this survey was designed to push respondents toward the conclusion most of them reached.
If the order in which the questions are reported by the journalist who wrote this story reflects the order in which the questions were actually asked, it would tend to lead respondents to the conclusion that, gee whiz, we’re just going to have to pay more. We might as well get used to the idea.
- Are Boomers are going to cause access to go down? (Gee, I guess so…)
- Are seniors going to have to pay more? (Golly, sure sounds like it!)
- Are you ready to acknowledge reality? (Ummm, yeah, I suppose…)
Canadians might actually think what this poll says they do. But, like the old song says, it ain’t necessarily so.
This post also appears on Rabble.ca.