A Comparison of Web and Mail Survey Response Rates
Michigan State University
 |
Introduction |
|---|
 TOP Introduction MethodsResultsConclusionReferences |
|---|
The Internet (also called the World Wide Web or the Web) is increasingly looked at as a means of surveying the public (Couper 2000). Possible advantages of using the Internet include cost savings associated with eliminating the printing and mailing of survey instruments (Cobanoglu, Warae, and Morec 2001) as well as time and cost savings of having returned survey data already in an electronic format. For special populations that regularly use the Internet, the Web has been found to be a useful means of conducting research (Couper, Traugott, and Lamias 2001; Sills and Song 2002). In some instances, a mixed-mode strategy has been suggested as a means for exploiting the advantages of Web surveys and minimizing nonresponse (Dillman 2000; Schaefer and Dillman 1998). To reliably use a mixed-mode strategy (e.g., mail surveys and Web surveys) or to select among alternative survey modes, researchers must understand and demonstrate the equivalency and complementarity, or relative strengths of alternative modes (Dillman 2000). Researchers have used survey response rates as one measure of equivalency.
Some studies suggest that in populations with access to the Internet, response rates for e-mail and Web surveys may not match those of other survey methods (Cook, Heath, and Thompson 2000; Couper 2000). Apparent differences in response rates for Web surveys and mail surveys have many causes or explanations. One explanation for these differences in response rates may be the fact that less time and attention have been devoted to developing and testing motivating tools to increase Web survey response, compared to the time spent studying tools employed in mail surveys (e.g., the use of personalization, precontact letters, follow-up postcards, and incentives). The widely followed elements of the "tailored design method" for mail surveys (Dillman 2000) were the product of years of research and intensive study. However, the implementation approaches that are beneficial for mail surveys may not translate directly to response rate benefits for Web surveys (Couper 2000). For example, research has revealed concerns on the part of potential survey participants that are particularly salient for Web users, including Internet security and the receipt of electronic "junk mail" or "spam" (Sills and Song 2002).
The research reported here examines the effect of surface mail contacts on Web survey response rates. We also examine the relative merit of using a mail survey in a population that has ready access to the Web. The reported research is based on a larger research effort at Michigan State University (MSU). In 2000, MSU commissioned campus researchers and staff to develop a watershed plan that would comply with the storm water management requirements of Phase II of the federal Clean Water Act (Witter et al. 2001). One part of this integrated research, teaching, and outreach effort was a campuswide survey (N = 19,890) of MSU students’ watershed knowledge, perceptions, and use.
|
Methods |
|---|
|
TOPIntroductionMethodsResultsConclusionReferences |
|---|
All students at MSU have an e-mail account and free access to the Internet. Furthermore, MSU students are expected to use the Internet to communicate with instructors and administrators, register for classes, and participate in their courses. We used an iterative design process to develop a watershed knowledge, attitude, and use questionnaire. The questionnaire was implemented using a hard copy mailed to a random sample of students and a Web questionnaire emailed to another random sample of students. The Web questionnaire was virtually an exact copy of the written instrument, except that students scrolled down and "clicked" to fill in their responses to the closed-ended questions instead of turning pages and using a pencil or pen to fill in ScanTron ® "response bubbles." Altogether, the survey contained 10 brief sections of questions with a total of 56 items to be completed. The Web instrument and the hard copy questionnaire each took about 15 minutes to complete.
The Web contact modes were structured so as to examine the relative benefits of multiple contacts on Web survey response rates within the university’s limitation of only one e-mail contact. It has been demonstrated that increased numbers of contacts result in increases in response rates, with prenotice contact appearing to have the strongest response rate impact (e.g., Dillman 2000; Dillman, Clark, and Sinclair 1995). Although some studies of electronic surveys show increased response rates from increased e-mail contacts (e.g., Mehta and Sivadas 1995; Smith 1997), only one study appears to have touched on the use of surface mail prenotice, paper questionnaires, and/or surface mail reminder contacts with electronic surveys (Schaefer and Dillman 1998). Schaefer and Dillman (1998) used a multimode approach (N = 904) to obtain responses from individuals who preferred written questionnaires or who were unreachable through e-mail. They found e-mail prenotice more effective in increasing response rates to an e-mail questionnaire than surface mail prenotice, and that paper questionnaires, all else being equal, seemed to enjoy higher response rates than e-mail surveys. Given the large sample size of student e-mail users at MSU, the costs of surface mail survey implementation, and the strict limitations on e-mail contacts (only one e-mail contact is allowed), we decided to examine whether surface mailing prenotices and/or reminders might significantly increase e-mail survey response rates. Furthermore, we wished to explore whether the additional surface mail contacts were worth the additional time and expense. Finally, we wanted to test the relative benefits of a multiple contact, hard copy survey versus an electronic survey for a population with complete access to the Internet.
The target population for the study was MSU undergraduate, graduate, and professional students enrolled for academic year 2001–2002. The MSU Registrar’s Office agreed to provide us with five random samples of students (their names together with mailing and e-mail addresses) drawn from the university’s current enrollment records—one list for the hard copy survey mode and four lists for each of the Web implementations. The sample lists were drawn to contain the same proportion of undergraduate, graduate, and professional students, were stratified to reflect the same proportionate distribution across MSU’s colleges, and were mutually exclusive. Because of financial constraints (i.e., printing and mailing costs), the sample size for the hard copy mode was to be 3,000, while the sample size for each of the four Web groups could be about 4,500. An error in the registrar’s office resulted in a hard copy sample of 2,594 and four Web-mode samples totaling 17,296.1
During November 2001 a total of 19,890 MSU students received an MSU watershed survey by either e-mail or U.S. mail. Recipients were divided into five groups differentiated by survey distribution mode. Within the constraints imposed by the university and the experimental design, we adopted a Dillman (2000) "tailored design" approach to implement the survey. Group 1 (mail, N = 2,594) received four contacts: a preliminary postcard, a hard copy survey with cover letter explaining the purpose of the study, a follow-up/reminder postcard, and a replacement hard copy survey with cover letter to nonrespondents. Table 1 illustrates the 2 x 2 factorial design of the Web implementation modes. Group 2 (Postcard/email, N = 4,327) received two contacts: a preliminary postcard and an e-mail with the same explanation of purpose as group 1’s cover letter and a hyperlink to the Web version of the survey. Group 3 (postcard/e-mail/postcard, N = 4,178) received three contacts: the preliminary postcard, the e-mail (as described above), and a follow-up/reminder postcard. Group 4 (e-mail/Postcard, N = 4,351) received two contacts: first the e-mail and then a follow-up postcard. The follow-up postcards were mailed 10 days after the delivery of the survey and only to those individuals that had not yet responded. Group 5 (e-mail, N = 4,440) received only one contact: the e-mail containing the explanation and the link to the survey.
|
Returned surveys were scored as responses if they were completed or partially completed. The response rate used for the analysis was calculated as the number of surveys returned divided by the number of surveys that were sent out and not returned as undeliverable. This response rate is the maximum response rate (RR6) as defined by the American Association for Public Opinion Research (AAPOR 2000). For each distribution mode, an item nonresponse rate was also defined as the average number of unanswered items per survey returned. There were no significant differences between modes for item nonresponse, and no further discussion of this portion of the analysis appears in this article.
|
Results |
|---|
|
TOPIntroductionMethodsResultsConclusionReferences |
|---|
The responses on key watershed survey variables from the five samples were roughly equivalent with one interesting divergence. The mean age of respondents in the four Web groups of 24.14 years old differed significantly, at the 0.05 level, from the 30.55 years old mean age of mail survey respondents. At the time of the survey, MSU data show that 80 percent of MSU students were 24 years of age or younger. Grouping the survey data on respondents’ age into the MSU age categories (
24 years old and >24 years old) revealed that in the aggregate about 70 percent of respondents were 24 years of age or younger. The observed age/mode response difference for e-mail versus paper surveys has not been widely reported or studied. While this survey’s substantive content did not seem significantly affected by age/mode-related differences, the study’s finding of significant mode response differences based on respondents’ age merits further study. Despite this age difference in the proportion of electronic and mail surveys returned, the five survey treatments revealed substantially similar information regarding the questionnaires’ substantive items. The study’s large sample size tends to make even small differences statistically significant. However, where there were statistically significant differences in mean responses to substantive variables among the modes, they were small in absolute size and not consequential for our interpretation or for use of the data for watershed planning and analysis. Analysis of the survey findings by mode (e.g., percentage of males, academic level, proximity to campus, use of the Red Cedar area) supports the notion that the five data sets are substantially equivalent.
Figure 1 shows the response rates (RR6), the standard deviations, the number of participants receiving survey questionnaires, and the production cost per completed questionnaire for each of the five distribution treatments. The largest response rate difference was between the mail and the e-mail only distribution modes, about 10 percentage points. The cost figures include the printing of survey booklets and postcards, envelopes, outgoing and return postage, as well as computer programming and hosting costs for the Web survey. These cost estimates use MSU’s "off-campus/non-faculty" computer programming and hosting costs for each of the Web modes to make them comparable. The cost estimates do not include survey design, evaluation, pretesting, and data entry costs as these costs were roughly equal across implementation modes. Excluding such expenses, each completed mail (group 1) questionnaire costed nearly $11 while the various Web survey treatments each costed less than $2.
|
Statistical analyses of the response rate data were used to test two research questions: (1) Do the different distribution treatments result in significant response rate differences? and (2) What, if any, are the main effects and interactions of pre- and post-survey contact treatments on response rates?
A one-way analysis of variance revealed a significant effect of treatment on response rate (F = 35.961, df’s = 4 and 19,885, p < .001, 
= .085). The nonlinear correlation (.085), however, indicated that the differences among response rates were relatively small.
To further explore the effect of implementation mode on response rates, analyses were undertaken to test specific differences among the individual methods and to estimate confidence intervals. Despite the large number of observations, the results show that group 1 (mail; RR = .315) and group 2 (postcard, e-mail; RR = .297) did not have significantly different response rates at the .05 significance level, nor were response rate differences between group 2 (Postcard, E-mail; RR = .297) and group 3 (Postcard, E-mail, Postcard; RR = .286) found to be significant. Other pair-wise comparisons, including the difference between group 1 (mail) and group 3 (postcard, e-mail, postcard) and between group 1 (mail) and group 5 (e-mail), revealed significant differences in response rates.
To examine the impact of the surface mail pre- and post-survey contact on Web survey response rates, we made use of the project’s 2 x 2 experiment-within-an-experiment design. The test results show that there is a clear and statistically significant separation between groups that received pre-survey postcards and groups that did not receive pre-survey postcards (F = 84.40, df = 1, p < .05). The second factor analyzed the impact on Web survey response rates for groups that received post-survey postcards and those that did not receive post-survey postcards. The test results show a significant and positive—though considerably smaller—main effect on response rates for groups receiving post-survey postcards (F = 7.052, df = 1, p < .05). Finally, the interaction between the two independent variables was statistically significant (F = 18.80, df = 1, p < .05). The positive effect of the reminder postcard was restricted to those who did not receive a prenotice card.
|
Conclusion |
|---|
|
TOPIntroductionMethodsResultsConclusionReferences |
|---|
This study found that a Web survey application achieved a comparable response rate to a mail hard copy questionnaire when both were preceded by an advance mail notification. A reminder mail notification had a positive effect on response rate for the Web survey application compared to a treatment in which respondents only received an e-mail containing a link to the Web survey. Reminder mail notifications did not produce higher response rates to the Web survey for respondents who had received a prenotice. The cost differential between the mailed hard copy questionnaire treatment and the Web survey treatments with mailed advance notice was substantial.
The findings of this research suggest that, in a population in which each member has Web access, a Web survey application can achieve a comparable response rate to a questionnaire delivered by surface mail if the Web version is preceded by a surface mail notification. A caveat is that we found a significant age difference in response to mail and Web survey versions. Further, considering Web survey applications alone, the findings suggest that a mail prenotice can increase response rates. In this study, a reminder notification was less effective. The cost advantage of a mail notification/Web questionnaire delivery combination suggests that this approach may be beneficial for studying populations with full access to the Internet. Further research comparing this approach with advance e-mail notification is needed. In addition, further research examining possible demographic differences in compliance with mail and Web survey requests is desirable.
|
Footnotes |
|---|
1. A student laborer’s key-punch error resulted in the smaller sample sizes, but that error did not impact the stratified, random nature of the sample lists. Before using the sample lists, we verified that the sample lists were random, mutually exclusive, and properly stratified.
|
References |
|---|
|
TOPIntroductionMethodsResultsConclusionReferences |
|---|
American Association for Public Opinion Research (AAPOR). 2000. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. Ann Arbor, MI: AAPOR.
Cobanoglu, Cihan, Bill Warde, and Patrick J. Moreo. 2001. "A Comparison of Mail, Fax, and Web Survey Methods". International Journal of Market Research 43:441–52.
Cook, Colleen, Fred Heath, and Russel L. Thompson. 2000. "A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys". Educational and Psychological Measurement 60:821–26.
Couper, Mick P. 2000. "Web Surveys: A Review of Issues and Approaches". Public Opinion Quarterly 64:464–94.[CrossRef][Web of Science][Medline]
Couper, Mick P., Michael W. Traugott, and Mark J. Lamias. 2001. "Web Survey Design and Administration". Public Opinion Quarterly 65:230–53.[Abstract]
Dillman, Don A. 2000. Mail and Internet Surveys: The Tailored Design Method. New York: Wiley.
Dillman, Don A., Jon R. Clark, and Michael A. Sinclair. 1995. "How Prenotice Letters, Stamped Return Envelopes, and Reminder Postcards Affect Mailback Response Rates for Census Questionnaires". Survey Methodology 21:1–7.
Mehta, Raj, and Eugene Sivadas. 1995. "Comparing Response Rates and Response Content in Mail Versus Electronic Mail Surveys." Journal of the Market Research Society 37:429–39.[Web of Science]
Schaefer, David R., and Don A. Dillman. 1998. "Development of a Standard E-Mail Methodology." Public Opinion Quarterly 62:378–97.[CrossRef][Web of Science]
Sills, Stephan J., and Chunyan Song. 2002. "Innovations in Survey Research: An Application of Web Surveys." Social Science Computer Review 20:22–30.
Smith, Christine B. 1997. "Casting the Net: Surveying an Internet Population." Journal of Communication Mediated by Computers 3 (1): available online at http://www.ascusc.org/jcmc/vol3/issue1/.
Witter, Scott G., Ruth Kline-Robach, David L. Long, Jon Bartholic, and Fred Poston. 2001. "MSU-WATER: A New Way of Addressing Water Quality Challenges." Water Resource Update 119:47–59.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  C. Greenlaw and S. Brown-Welty A Comparison of Web-Based and Paper-Based Survey Methods: Testing Assumptions of Survey Mode and Response Cost Eval Rev, October 1, 2009; 33(5): 464 - 480. [Abstract] [PDF]
|
|
|
|
 |
|
 D. J. Schleicher, R. A. Bull, and S. G. Green Rater Reactions to Forced Distribution Rating Systems Journal of Management, August 1, 2009; 35(4): 899 - 927. [Abstract] [PDF]
|
|
|
|
 |
|
 V. Jadva, T. Freeman, W. Kramer, and S. Golombok The experiences of adolescents and adults conceived by sperm donation: comparisons by age of disclosure and family type Hum. Reprod., August 1, 2009; 24(8): 1909 - 1919. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. Peytchev Survey Breakoff Public Opin Q, April 2, 2009; (2009) nfp014v1. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 L. Kahlor and S. Rosenthal If We Seek, Do We Learn?: Predicting Knowledge of Global Warming Science Communication, March 1, 2009; 30(3): 380 - 414. [Abstract] [PDF]
|
|
|
|
|
|
T. Freeman, V. Jadva, W. Kramer, and S. Golombok Gamete donation: parents' experiences of searching for their child's donor siblings and donor Hum. Reprod., March 1, 2009; 24(3): 505 - 516. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. A. Taylor, C. Ishida, and D. W. Wallace Intention to Engage in Digital Piracy: A Conceptual Model and Empirical Test Journal of Service Research, February 1, 2009; 11(3): 246 - 262. [Abstract] [PDF]
|
|
|
|
|
|
D. Heerwegh and G. Loosveldt Face-to-Face versus Web Surveying in a High-Internet-Coverage Population: Differences in Response Quality Public Opin Q, December 1, 2008; 72(5): 836 - 846. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 K. A. BARCHARD and J. WILLIAMS Practical advice for conducting ethical online experiments and questionnaires for United States psychologists Behav Res Methods, November 1, 2008; 40(4): 1111 - 1128. [Abstract] [PDF]
|
|
|
|
 |
|
 C. A. Klofstad, S. Boulianne, and D. Basson Matching the Message to the Medium: Results from an Experiment on Internet Survey Email Contacts Social Science Computer Review, November 1, 2008; 26(4): 498 - 509. [Abstract] [PDF]
|
|
|
|
 |
|
 M. L. Ybarra, M. Diener-West, D. Markow, P. J. Leaf, M. Hamburger, and P. Boxer Linkages Between Internet and Other Media Violence With Seriously Violent Behavior by Youth Pediatrics, November 1, 2008; 122(5): 929 - 937. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 I. Grava-Gubins and S. Scott Effects of various methodologic strategies: Survey response rates among Canadian physicians and physicians-in-training Can Fam Physician, October 1, 2008; 54(10): 1424 - 1430. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. S. Chima, C. Dietz-Seher, and S. Kushner-Benson Nutrition Risk Screening in Acute Care: A Survey of Practice Nutr Clin Pract, August 1, 2008; 23(4): 417 - 423. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. Denscombe The Length of Responses to Open-Ended Questions: A Comparison of Online and Paper Questionnaires in Terms of a Mode Effect Social Science Computer Review, August 1, 2008; 26(3): 359 - 368. [Abstract] [PDF]
|
|
|
|
|
|
M. L. Ybarra and K. J. Mitchell How Risky Are Social Networking Sites? A Comparison of Places Online Where Youth Sexual Solicitation and Harassment Occurs Pediatrics, February 1, 2008; 121(2): e350 - e357. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. Siikamaki and K. Wernstedt Turning Brownfields into Greenspaces: Examining Incentives and Barriers to Revitalization Journal of Health Politics Policy and Law, January 1, 2008; 33(3): 559 - 593. [Abstract] [PDF]
|
|
|
|
|
|
S. R. Porter and M. E. Whitcomb Mixed-Mode Contacts In Web Surveys: Paper is Not Necessarily Better Public Opin Q, December 1, 2007; 71(4): 635 - 648. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. Sanders, H. D. Clarke, M. C. Stewart, and P. Whiteley Does Mode Matter For Modeling Political Choice? Evidence From the 2005 British Election Study Political Analysis, July 1, 2007; 15(3): 257 - 285. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. J. Davis Consumers' Preferences For The Communication Of Risk Information In Drug Advertising Health Aff., May 1, 2007; 26(3): 863 - 870. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
B. Healey Drop Downs and Scroll Mice: The Effect of Response Option Format and Input Mechanism Employed on Data Quality in Web Surveys Social Science Computer Review, February 1, 2007; 25(1): 111 - 128. [Abstract] [PDF]
|
|
|
|
 |
|
 K. Kim, G. Jogaratnam, and J. Noh Travel decisions of students at a US university: Segmenting the international market Journal of Vacation Marketing, October 1, 2006; 12(4): 345 - 357. [Abstract] [PDF]
|
|
|
|
|
|
M. Denscombe Web-Based Questionnaires and the Mode Effect: An Evaluation Based on Completion Rates and Data Contents of Near-Identical Questionnaires Delivered in Different Modes Social Science Computer Review, May 1, 2006; 24(2): 246 - 254. [Abstract] [PDF]
|
|
|
|
|
|
S. R. Porter and M. E. Whitcomb E-mail Subject Lines and Their Effect on Web Survey Viewing and Response Social Science Computer Review, August 1, 2005; 23(3): 380 - 387. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||