AUGUST 18, 2012

How do cell phones affect telephone surveying?

A 2010 study recently published by Statistics Canada revealed that an increasing proportion of households are preferring cell phones to landlines: 13% of respondents reported using a cell phone exclusively, compared to 8% in 2008. The shift is particularly significant for younger households: half of the households aged 18-34 relied only on cell phones, a considerable increase from 34% in 2008.

The study also reported that about two thirds (68%) of exclusive cell phone users were renters rather than homeowners. On the flipside, in 2006, the same proportion of Canadians overall reported owning their home.

However, the implications of these findings in terms of representativeness remain unclear — we could not identify any Canadian studies that analyzed the characteristics of cell phone-only households in comparison to landline and mixed households. Even if this information existed, it is possible that such studies would have already been dated, given the high rate of change.

Measurement and Representativity

Some information is available from the US, however. The American Association for Public Opinion Research (AAPOR) Cell Phone Task Force published a paper in 2008 that points to two main measurement issues related to cell phone surveying. "First, there currently is no reliable evidence that the data gathered in good cell phone surveys are of lower quality than in comparable landline surveys. However, the Task Force believes it is advisable that researchers remain attentive to this concern. (...) Second, many new survey items may be needed for use in adjusting phone samples prior to analyzing their data. (...) the reliability of these new items has not yet been established." (page 5) Another report by the same AAPOR Task Force published in 2010 has adopted a more positive outlook on using cell phone surveying and representativeness: given the particularities of cell phone-only users in the United States (page 6 of the report mentions young adults, males, minorities, etc.), it concludes that using cell phones in a random digital dialing (RDD) frame "is very good for telephone survey researchers in terms of reaching more representative unweighted samples of the general public." (page 6) Nonetheless, issues with how to handle mixed households reached on cell phones as well as how to optimally screen for cell phone use and interview appropriateness remain unresolved.

Survey Costs

A paper published by the Canadian Marketing Research and Intelligence Association (MRIA) in 2007 predicts drastically increased surveying costs due to various factors linked with cell phone-only populations: need for an incentive, lower response rate, need to place more initial calls (because of more terminations linked with underage participants, busy or distracted respondents, etc.), need to abide by cell phone etiquette and leave a message and call-back number. According to MRIA, these factors, among others, would mean surveying costs 2.4 times higher for cell phone populations than for landline users (page 13); the AAPOR concurs, suggesting that the cost of cell phone surveys are generally between 2 and 4 times more expensive than landline ones (2010, page 11).

The 2010 AAPOR has drawn on experience from recent years to make a number of suggestions about cell phone surveying etiquette. Recommendations include:

• Calling during evenings and weekend, when a large proportion of cell phone users would not pay for the minutes used up by the survey, to mitigate the problem of participants paying for air time they did not wish to spend in the first place.
• Implementing more thorough geographic and other eligibility screenings. For instance, "if the area to be covered by the survey has a well-known and well understood name (...), the question that is asked of those reached on their cell phones is straight-forward, although it should be worded in a manner that does not tip off the person being spoken to as to what answer will qualify or disqualify the potential respondent to be interviewed." (page 85).
• Allowing a cell phone to ring at least eight times (compared to six rings for landlines) before filing the call as "Ring No Answer (RNA)" (page 87), and, as mentioned earlier, leaving a detailed message with a call-back number when directed to a voicemail.
• Being prepared to schedule a call back, given that respondents might be distracted or unable to speak immediately or in an unsafe situation when reached (page 89).
• Adapting the interviewer training for cell phone surveys (page 92).

The AAPOR remains optimistic, and concludes its 2008 report by stating that "ongoing research has demonstrated that conducting survey interviews by cell phone users in the U.S. is feasible, if also somewhat difficult and more expensive than conducting similar interviews on landline telephones." (page 3) The 2010 AAPOR Task Force report makes three recommendations, all of which are linked with researchers acting purposefully: "(1) researchers should explain the method by which the cell phone numbers used in a survey were selected, (2) if RDD telephone surveys do not sample cell phone numbers, then researchers should provide an explanation of how excluding cell phone owners might or might not affect the survey's results, and (3) researchers should explain the decisions that were made concerning weighting of cell phone samples, including why the sample was not weighted, if in fact that was the case." (page 11)

Ethical Considerations

The ethics of cell phone surveying should also be taken into consideration. The AAPOR Task Force, in its 2010 report, addresses this question and makes a number of suggestions. Recommendations include being mindful of the possibility that respondents not be in the time zone indicated by their area code (page 79), that special attention be paid to the frequency and timing of calling attempts, that detailed caller identification be disclosed (also to help increase the number of call-backs, as mentioned earlier), and that the survey providers maintain an internal "Do Not Call List", to minimize the chances of repeated burden. Moreover, the AAPOR Task Force suggests that special attention be paid to the safety and privacy of respondents. Over and above ensuring (potentially at the start of the survey and part-way through) that the respondent is not distracted by another activity (including, but not limited to, driving, walking, etc.), the survey provider should ensure the respondent's privacy by wording questions so that "respondents in public or semi-private places should not be required to verbalize responses that could (1) reasonably place them at risk of criminal or civil liability; (2) be damaging or to their financial standing, employability, or reputation; or (3) otherwise violate their privacy" (page 81).

 Return to the home page
 List all news