Pollster questions accuracy of online sampling

Barb Sweet
Send to a friend

Send this article to a friend.

Don Mills — Telegram Photo

A pollster is questioning the reliability of online polling after some predictions prior to Tuesday’s election were off.

“There is a big discussion in our industry and a debate going on,” said Don Mills, president and CEO of Corporate Research Associates (CRA).

“There’s a lot of people who say online research is just as good as telephone research. That has not been proven to be true and we have recent examples in Atlantic Canada where a competitor of ours has used an online methodology and have not got it within the margin of error they quoted,” he said.

“They are not even supposed to quote margin of error in online polls.”

An in-depth poll by CRA conducted for The Telegram came closest to election night results, Mills said.

“We are very proud of our record. We have been doing this for more than 25 years.

“Every time we put a prediction out there, we put our reputation on the line. And we always get criticized by one party or another for being not representative of what is actually happening.”

Mills said telephone polling is still the most accurate method when it comes to predicting election results.

CRA, using a telephone poll based on a sample size of 800, predicted 59.5 per cent support for the PCs among decided voters, 24.7 per cent for the NDP and 15.8 per cent for the Liberals.

The actual election results were 56.1 per cent for the PCs; 24.6 for the NDP; and 19.1 per cent for the Liberals — a total difference of 6.8 per cent from the poll prediction.

When leaning voters were factored in, the CRA prediction was even closer — with 56.7 per cent support for the PCs, 26.4 for the NDP and 16.9 for the Liberals.

CRA quoted a margin of error of plus or minus 4.1 per cent on decided voters and plus or minus 3.7 per cent on leaning and decided voters.

A poll by MQO Research conducted between Sept. 28-30 pegged support for the Liberals among decided voters as being 13 per cent.

Based on a sample of 464 residents, the survey was conducted via phone and online.

Some 54 per cent of those who had made up their mind backed the Kathy Dunderdale-led PCs.

New Democrat support in the MQO poll was pegged at 33 per cent among the decided.

MQO cited a margin of error of plus or minus 4.6 per cent. 

“They used an online methodology which is not as accurate, frankly — obviously,” Mills said.

Environics did online polling for The Canadian Press, which included a disclaimer that “the non-random nature of online polling makes it impossible to determine statistically how accurately the results reflect the opinions of the population at large.”

The Environics poll predicted 38 per cent of respondents would back the Progressive Conservatives, compared to 23 per cent for the NDP and nine per cent for the Liberals.

Mills said online sampling doesn’t represent all demographics, especially low-income earners and seniors,  who may not have access to computers.

Online research may be more valuable for marketing campaigns where feedback is sought, he said, but it’s highly unreliable when it comes to elections.

MQO Research did not respond to a request for comment.

Mills said MQO may be violating the codes and conducts of the Marketing Intelligence and Research Association (MRIA) when it associates margin of error with polls that include online research.

According to its website, MRIA’s code of conduct and good practice prohibits “statements about margins of sampling error or population estimates with regard to convenience samples, including most online panels.”

But MRIA executive director Brendan Wycks said because MQO declared it used the telephone to randomly recruit people and invite them to the online survey, it did not technically violate the code of conduct.

He acknowledged Mills is questioning if the survey was actually done that way.

Wycks said if online polling is done rigorously and by a reputable company, it can be remarkably reliable, even in elections.

But he said the issue of using online and emerging technologies is stirring debate in the industry, and the association’s standards committee will meet next week to discuss whether or not the code of conduct needs to be more precise.

Pollsters are facing more pressure from their clients to do more and be more economical.

Especially daunting for the association is the speed of development of new technology and social media over the past five years.

Wycks said the association relies on sister groups around the world to ensure standards are consistent globally.

Online polling is cheaper than telephone polling. Even cheaper is a method in which surveys are automated — the calls are dialed randomly but there is no human asking questions and the respondent replies by pressing touch tone buttons on their phone. But that raises the issue of who’s pushing the buttons — the respondent could be a child.

Still, Wycks said research by firms that use that method and studies have suggested that it can be reliable.



Organizations: Corporate Research Associates, NDP, The Telegram MRIA MQO Research Canadian Press Progressive Conservatives Marketing Intelligence and Research Association

Geographic location: Atlantic Canada

  • 1
  • 2
  • 3
  • 4
  • 5

Thanks for voting!

Top of page



Recent comments

  • Kate Bastow
    October 16, 2011 - 09:42

    This is not surprising. The more hurdles you put in place, the more biased your sample will become, as you are going to filter out the particular group for which the hurdle is an issue. i.e., if you phone and then direct someone to an online survey, you've first got to get past the 'staying on the line and listening' hurdle, and then the 'continuing to the website" hurdle, and then the 'finish filling out the survey' hurdle. I had no idea the polls were so accurate. Everyone should just exercise their right to keep their vote secret and then this debate is a non-issue :-)

  • Moe St.Cool
    October 16, 2011 - 08:39

    Turns out Aylward was right after all. M5's poll was skewed and leaned improperly towards the PC's, hence giving an unfair advantage. Also, look at the sample size, 464 (vs the 800 used by CRA), those numbers are HUGE statistically speaking. M5 should be investigated over their lazy/unethical tactics.

  • just sayno
    October 15, 2011 - 10:54

    Based on the assumption that the pollsters are paid to collect data, the employee calling the respondents between 5:00 and 8:00pm are compensated, my question to the pollster is why should the respondent spend time providing the data for no compensation? Until pollsters compensate for the data provided my suggestion is to tell the caller who is intruding on your personal time nothing or the opposite of how you feel on any given subject.

  • Harvey
    October 15, 2011 - 10:00

    Really...what does polling prove that is beneficial to the average voter? I changed my vote this time and no amount of polling was going to change my mind. Sometimes they look like childish games!!!