Pollster questions accuracy of online sampling

A pollster is questioning the reliability of online polling after some predictions prior to Tuesday’s election were off.

“There is a big discussion in our industry and a debate going on,” said Don Mills, president and CEO of Corporate Research Associates (CRA).

“There’s a lot of people who say online research is just as good as telephone research. That has not been proven to be true and we have recent examples in Atlantic Canada where a competitor of ours has used an online methodology and have not got it within the margin of error they quoted,” he said.

“They are not even supposed to quote margin of error in online polls.”

An in-depth poll by CRA conducted for The Telegram came closest to election night results, Mills said.

“We are very proud of our record. We have been doing this for more than 25 years.

“Every time we put a prediction out there, we put our reputation on the line. And we always get criticized by one party or another for being not representative of what is actually happening.”

Mills said telephone polling is still the most accurate method when it comes to predicting election results.

CRA, using a telephone poll based on a sample size of 800, predicted 59.5 per cent support for the PCs among decided voters, 24.7 per cent for the NDP and 15.8 per cent for the Liberals.

The actual election results were 56.1 per cent for the PCs; 24.6 for the NDP; and 19.1 per cent for the Liberals — a total difference of 6.8 per cent from the poll prediction.

When leaning voters were factored in, the CRA prediction was even closer — with 56.7 per cent support for the PCs, 26.4 for the NDP and 16.9 for the Liberals.

CRA quoted a margin of error of plus or minus 4.1 per cent on decided voters and plus or minus 3.7 per cent on leaning and decided voters.

A poll by MQO Research conducted between Sept. 28-30 pegged support for the Liberals among decided voters as being 13 per cent.

Based on a sample of 464 residents, the survey was conducted via phone and online.

Some 54 per cent of those who had made up their mind backed the Kathy Dunderdale-led PCs.

New Democrat support in the MQO poll was pegged at 33 per cent among the decided.

MQO cited a margin of error of plus or minus 4.6 per cent. 

“They used an online methodology which is not as accurate, frankly — obviously,” Mills said.

Environics did online polling for The Canadian Press, which included a disclaimer that “the non-random nature of online polling makes it impossible to determine statistically how accurately the results reflect the opinions of the population at large.”

The Environics poll predicted 38 per cent of respondents would back the Progressive Conservatives, compared to 23 per cent for the NDP and nine per cent for the Liberals.

Mills said online sampling doesn’t represent all demographics, especially low-income earners and seniors,  who may not have access to computers.

Online research may be more valuable for marketing campaigns where feedback is sought, he said, but it’s highly unreliable when it comes to elections.

MQO Research did not respond to a request for comment.

Mills said MQO may be violating the codes and conducts of the Marketing Intelligence and Research Association (MRIA) when it associates margin of error with polls that include online research.

According to its website, MRIA’s code of conduct and good practice prohibits “statements about margins of sampling error or population estimates with regard to convenience samples, including most online panels.”

But MRIA executive director Brendan Wycks said because MQO declared it used the telephone to randomly recruit people and invite them to the online survey, it did not technically violate the code of conduct.

He acknowledged Mills is questioning if the survey was actually done that way.

Wycks said if online polling is done rigorously and by a reputable company, it can be remarkably reliable, even in elections.

But he said the issue of using online and emerging technologies is stirring debate in the industry, and the association’s standards committee will meet next week to discuss whether or not the code of conduct needs to be more precise.

Pollsters are facing more pressure from their clients to do more and be more economical.

Especially daunting for the association is the speed of development of new technology and social media over the past five years.

Wycks said the association relies on sister groups around the world to ensure standards are consistent globally.

Online polling is cheaper than telephone polling. Even cheaper is a method in which surveys are automated — the calls are dialed randomly but there is no human asking questions and the respondent replies by pressing touch tone buttons on their phone. But that raises the issue of who’s pushing the buttons — the respondent could be a child.

Still, Wycks said research by firms that use that method and studies have suggested that it can be reliable.

bsweet@thetelegram.com