Public Opinion Quarterly Advance Access originally published online on December 9, 2008
Public Opinion Quarterly 2008 72(5):962-984; doi:10.1093/poq/nfn061
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
This article appears in the following Public Opinion Quarterly issue: Special Issue: Web Survey Methods [View the issue table of contents]
Does a Probability-Based Household Panel Benefit from Assignment to Postal Response as an Alternative to Internet-Only?
Address correspondence to Bryan Rookey; e-mail: bdrookey{at}wsu.edu.
 |
Abstract |
|---|
 TOP Abstract IntroductionBackground for this ResearchData and MethodsResults and FindingsDiscussion and ConclusionsReferences |
|---|
A potential limitation of web-only panels of the general public, even when households are selected using probability methods, is that only about 70 percent of U.S. households have members with Internet access. In addition, some members of Internet-connected households may be unable or unwilling to participate over the web. The Gallup Panel uses both web and mail modes to survey respondents and in 2006 included approximately 50,000 households selected by random-digit dialing. Frequent Internet users were assigned to respond by the web, while others were assigned to participate by mail using a paper questionnaire with a similar visual layout to the web. We use several approaches to determine whether or not the mail option adds value to the results in an otherwise Internet panel and organize our analyses around answering a series of questions. First, does the use of mail allow different types of people to be included? Second, do mail and web respondents give different answers to the same questions? Third, does weighting on and controlling for demographics eliminate any differences in responses from mail and web respondents and indicate that mail is not needed? Finally, do differences exist when responses are collected using an independent mode? In general, the answers to these questions suggest that use of mail adds value to the panel results and improves the overall accuracy of survey results.
|
Introduction |
|---|
|
TOPAbstractIntroductionBackground for this ResearchData and MethodsResults and FindingsDiscussion and ConclusionsReferences |
|---|
In recent years, interest has developed for using Internet panels of individuals who periodically complete web surveys to estimate the opinions and behaviors of the general public. This interest in such panels stems from a number of considerations, including the existence of professional standards against contacting individuals via email unless there is a "preexisting" relationship (CASRO 2008) as well as the cost savings over telephone or mail for conducting cross-sectional general public surveys.
Two types of panels have emerged (see Couper 2000
). One is a nonprobability approach whereby advertisements are used to encourage people to volunteer to participate only if they are willing to complete surveys over the Internet. In addition to not having a probability base that will allow statistical generalization of results to a larger population, such panels are limited further because people without Internet access, currently about 30 percent of U.S. households (Pew 2007), are excluded from all surveys. Attempts to overcome these limitations are made through the extensive use of postsurvey adjustments, albeit with questionable success (Lee 2006a; Bethlehem and Stoop 2007).
The second type of Internet panel uses probability sampling methods with random-digit dialing (RDD) or in-person contact in order to provide nearly all households a known nonzero chance of being included, regardless of whether they have Internet access. People who agree to be surveyed are then asked to respond over the Internet. If they lack Internet access, they may be provided with such access and supported in their efforts to use it or asked instead to respond by telephone or mail. It is unclear at this time whether attempting to survey people who lack Internet access or are less able or willing to use it provides more accurate results than would be obtained by only surveying Internet users and then adjusting the results in an attempt to represent the entire survey population. If mail does not contribute to the results, then surveying only Internet users could realize substantial savings.
Our purpose in this paper is to examine results from a limited data collection period for a particular probability panel, the Gallup Panel, which recruits households through RDD and then assigns participants to either mail or web as a means of responding. Specifically, through a series of analyses we seek to determine whether value is added by contacting respondents who are unable or less willing to respond to the Internet, or whether alternatives such as weighting the Internet data will produce equivalent results.
We organize our investigation around answering a series of questions using one Gallup Panel survey and a related telephone poll, both of which were conducted in Fall 2006. First, we determine whether or not mail allows different types of people to be included in an otherwise Internet panel by comparing demographic characteristics between those who have access and use the Internet frequently and infrequent or non-Internet users who are assigned to respond by mail. Second, we examine whether mail-assigned and web-assigned respondents give different answers to the same survey questions. Third, we determine if weighting on and controlling for demographic characteristics eliminates any differences in responses from mail and web respondents and indicates that mail is not needed. Fourth, we use data collected from mail-assigned and web-assigned panelists by a neutral mode (telephone) to determine whether the inclusion of those unwilling or unable to respond by web improves the accuracy of election predictions.
|
Background for this Research |
|---|
|
TOPAbstractIntroductionBackground for this ResearchData and MethodsResults and FindingsDiscussion and ConclusionsReferences |
|---|
Demographic differences between mail and web respondents
Based on estimates of Internet user characteristics from the 2007 Pew Internet and American Life Project (2007
) and other studies that examine differences between web and mail respondents (McDevitt and Small 2002; NTIA 2004; Link and Mokdad 2006), it is quite likely that Internet users will differ substantially from those who do not use the Internet. According to Pew data at the time responses were collected for the present study, nearly equal proportions of males and females reported using the Internet (70 percent of females versus 71 percent of males). Internet users tended to be younger, with about 32 percent of people over the age of 65 using the Internet compared to 87 percent of 18–29 year olds. People with higher household incomes were also more likely to use the Internet than those with low household income; 93 percent of households reported $75,000 and above, compared to 55 percent of those with less than $30,000 per year of income. In addition, while 91 percent of people with college degrees reported using it, the percentage fell to 61 percent for those with only a high school education and 40 percent for those with less than a high school education. Similarly, Link and Mokdad (2006) found that web respondents to web and mail alternatives of the CATI-based Behavioral Risk Factor Surveillance System differed significantly in terms of gender, age, and income. Based on these data from Pew (2007) and related studies (Link and Mokdad 2006), it appears that providing an alternative means of responding to Internet panel surveys would allow people who have quite different demographic characteristics (than those of Internet users) to be included in panel survey results.
To examine whether using a mail response mode brings in people who are different from the web respondents, we compare the characteristics of people assigned to mail and web response modes in the Gallup Panel. In contrast to frequent Internet users, those assigned to mail either have no access, have access but do not use it, use it infrequently, or have asked to be surveyed by mail instead of the Internet. If the demographic composition of the mail group is substantially different from the composition of the web group, it would suggest that the use of mail allows different types of people to participate who cannot be surveyed effectively over the Internet. We expect that the characteristics of people assigned to receive mail versus web questionnaires would correspond to the demographics reported by the Pew Internet and American Life Project (2007) for users and nonusers, but it is important to determine whether or not that is the case.
Web versus mail answers for a selected survey
Excluding certain subgroups from a panel survey does not necessarily mean that the results will differ from what one might get by including them. It may be that answers to opinion and behavior questions in Gallup Panel surveys will not differ for people by age, income or educational attainment, for example. Thus, as an initial step in this analysis, it is important to compare answers received by web and the mail alternative. Unless answers differ, there would seem to be little benefit from offering mail questionnaires to people either without Internet access, or who have access, but tend not to use it. We address this issue by selecting one Gallup Panel survey, for which we examine all questions to determine the extent to which differences in responses exist between the mail and web respondents. If answers do not differ, assignment to postal response as an alternative to Internet-only in a probability-based household panel would not appear to add to the results, but if they do differ, the mail portion may be picking up an important nonrandom subgroup that would otherwise be excluded.
Weighting on and controlling for demographic characteristics
There has been extensive discussion in the literature on whether or not it is possible to make postsurvey adjustments to Internet-only samples to represent both Internet and non-Internet users (for examples of this discussion, see Kehoe and Pitkow 1996; Couper 2000; Witte, Amoroso, and Howard 2000; Duffy et al. 2005; Terhanian 2000; Sparrow and Curtice 2004; Lee 2006a; Couper et al. 2007). These procedures assume, for example, that someone who is older and has Internet access holds similar views to someone in the same age category who does not have Internet access. While there are benefits to poststratification (Biemer and Christ 2008), recent research suggests that poststratification adjustment of results obtained from only one segment of the population (e.g., Internet users who volunteer to participate in panels) are insufficient for most research purposes that seek to describe the opinions and behaviors of the public (Vehovar and Manfreda 1999; Lee 2006a, 2006b; Couper et al. 2007).
First, to simulate how final weighted estimates would differ with and without mail respondents, we compare weighted responses of the combined mail and web respondents with the separately weighted web responses. If replication of the results obtained by mail and web combined were possible through traditional weighting adjustments of only the web responses, this would suggest that data derived from web respondents selected by probability sampling and weighted to reflect the population may be sufficient to accurately estimate the opinions and behavior of the general public. In this case, there would appear to be little or nothing to gain from using mail as a second mode for the study analyzed here. However, if weighting web-only responses does not produce equivalent results to the mail and web combined responses, then including respondents who are not willing or less able to respond by web would, theoretically, reduce coverage bias. This would imply that both traditional weighting is insufficient (as argued by Lee 2006a, 2006b) and that mail adds something to the results that web-only designs would not reproduce.
Second, we examine the influence of demographic characteristics and mode of response in a different way to gain further insight into the potential influence of offering a mail option. Specifically, we use logistic regression techniques to hold demographic characteristics constant while estimating the effect of mode of response on respondents’ answers to specific survey questions. If mode does not have a statistically significant impact, this would indicate that using only the web data is likely to produce the same results as those achieved by including paper respondents, and therefore mail is not needed. However, if mode is a statistically significant predictor of answers after controlling for demographic characteristics, it could suggest that mode itself influences answers, or that traditional demographic characteristics may not eliminate mail versus web differences in answers to these questions.
Effect of mode choice on respondent answers
It is possible that answers to the many opinion and behavior questions asked in Gallup Panel surveys differ simply because the modes produce different answers, regardless of how or to what extent demographics of the web and mail groups diverge. The general conclusion suggested by prior research is that although the existence of differential mode effects from web and mail cannot be ruled out as the cause of any differences observed in this study, the use of a very similar visual appearance for questions in the paper and mail survey modes will help avoid the occurrence of intermode differences in Gallup Panel surveys (Dillman, Smyth, and Christian 2009). For this reason, the Gallup Panel has made explicit efforts to provide the same visual presentation to both web and mail respondents. However, to control for the potential impact of mode choice in this research, we surveyed both mail-assigned and web-assigned panelists via a third mode (telephone) during the same period as the World Affairs survey. We then compared the voting intentions of mail- and web-assigned panelists to the official election outcomes in the November 2006 general election.
|
Data and Methods |
|---|
|
TOPAbstractIntroductionBackground for this ResearchData and MethodsResults and FindingsDiscussion and ConclusionsReferences |
|---|
The gallup panel
The Gallup Panel uses RDD to select households from the U.S. general public. Although 15 percent of U.S. households do not have telephone landlines (Blumberg and Luke 2007
) and are outside the sampling frame, this is much lower than the 30 percent of households without web access. The Gallup Panel has worked to retain panelists who have abandoned landline service in recent years, so the coverage provided is somewhat greater than 85 percent. In the first step of the recruitment process, random-digit calls are made and the interviewer asks to speak to an adult member of the household. During this first telephone contact to potential panelists, information about the Gallup Panel is explained and the household respondent is asked if he or she is willing to enlist in the panel. Incentives in the form of money or prizes are not offered or promised to potential panelists at any time. All respondents who agree to participate are quickly sent a paper "welcome packet" via mail thanking them for joining the panel and including a welcome survey to enroll the individual and up to three other members of the household. The questionnaire contains questions about demographics, personal computers in the household, Internet access, Internet use, and a return envelope with prepaid postage is provided. By November 2006, the response rate of the initial panel recruitment telephone contact was 26 percent (AAPOR RR3), and approximately 55 percent of those persons who completed the recruitment interview consented to join the panel and completed the welcome package, making them eligible for assignment to specific panel surveys. The response rate is further reduced due to monthly attrition of 2–3 percent.
Gallup uses responses to questions about household Internet access and Internet use to assign respondents to answer future surveys via mail or web. If reported Internet use is twice per week or more and the respondents provided an email address, they were assigned to receive web questionnaires; respondents who do not use the Internet, or do so less than twice a week or did not provide an email address, are assigned to receive paper questionnaires by mail. About 48 percent (approximately 22,000) of all Gallup panelists were assigned to participate in the panel through receipt of web questionnaires and about 52 percent (approximately 24,000) were assigned to mail.
Because Gallup assigns respondents to modes in this way, the mail versus web comparisons in the survey described here do not correspond exactly to whether people have or do not have Internet access. We explored an alternative approach based on three groups: (A) those that have household access and use the Internet frequently; (B) those that have household Internet access, but do not use it, use it once a week or less often, did not provide an email address, or explicitly requested at the time the enrollment form was completed that they be surveyed only by mail; and (C) those without Internet access (see table 1). However, of the people in group B, approximately two-thirds have no email, did not provide an email address, never access the email account they had, or did so infrequently (once a week or less). Thus, it seems unlikely that most of these individuals could be convinced to respond by web, because they were not contacted via email, or they were infrequent (and perhaps less skilled) users of the Internet. The remaining one-third, although they provided an email address, asked that they receive questionnaires by mail. When we examined the demographic characteristics of this group (table 1), we found that their characteristics were intermediate between those assigned to web and those without Internet access who were assigned to mail. Although we could have divided and reclassified these individuals, putting the frequent users who provided addresses into the web group and kept the remainder in the mail group, we decided to keep all of these in the mail contact group for purposes of the analysis. Doing so, allows keeping the groups in this analysis consistent with the data collection decisions made by Gallup, which are arguably in line with the feasibility of obtaining a response by the two modes.1 In addition, the effect of this decision is to make our analysis somewhat more conservative by including a limited number of people who are Internet users but seem less likely to respond to the web.
|
Specific survey under examination
We selected one questionnaire2 sent to respondents in October 2006 for the comparisons made in this investigation. It was described to respondents as the "World Affairs survey" and contained 92 items on such topics as foreign aid spending, and views on several foreign countries and the United Nations, among others. Significant efforts were made to use the same visual layout for both paper and web questionnaires. The data collection period for this survey was complete shortly before the telephone poll (described later) was conducted (November 2006). The final stage response rate was 69 percent of those to whom it was sent. The response rate for web panelists was 72 and 66 percent for mail respondents (AAPOR RR3). The overall response rate for this survey was 10 percent and is calculated using percent of household telephone numbers contacted for inclusion in the panel (RDD 26 percent x RDD respondents who joined 55 percent x survey 69 percent). The overall response rate for this survey is affected by a monthly panel attrition of 2–3 percent.
Weighting methods
We examine whether weighting on demographic characteristics reveals that the use of mail contributes to final survey results in a way that surveys of only frequent web users cannot. The poststratification weighting procedure reduces bias from overrepresentation or underrepresentation of particular demographic categories in the sample and within the mail group or the web group that may artificially produce mail versus web response differences. This process involves two key aspects: the demographic composition of the sample and accurate information about the composition of the general population (Biemer and Christ 2008, p. 330). Gallup constructs weights to ensure that particular strata of the population are not overrepresented or underrepresented in the sample on the basis of age (18–24, 25–34, 35–44, 45–54, 55+), education (high school or less, some college, college graduate or more), Census region (Northeast, Midwest, South, West), gender (male, female), race (white, black, all other), and Hispanic ethnicity (Hispanic, non-Hispanic). We computed two sets of weights—the first set included mail and web respondents and the second set included web respondents only. The two sets of weights match what would be used for a survey of all panelists as well as weights that would be used for a survey of only web respondents.
Logistic regression methods
We use logistic regression techniques (see Long 1997; Pampel 2000) to estimate the effects of survey mode alone and while controlling for demographic characteristics on the probability of selecting a particular response option to several survey questions. This approach goes beyond typical and standard poststratification variables to consider a fuller set of respondent attributes and is not limited by trimming rules introduced to reduce variation. The dependent variables are the top 30 questionnaire items that mail and web answers differed by five or more percentage points (as displayed in table 4). Responses to each question were recoded so that the dependent measures are dichotomous variables. We included 10 demographic control variables3 listed in table 2. All of the demographic information on panelists was collected at previous points but the information was continually updated throughout the panelists’ tenure. To compare across models, cases with incomplete demographic information were removed (n = 4,819).
|
|
Telephone comparison data for web-assigned and mail-assigned panelists
Active members of the Gallup Panel were polled by telephone two days before the November 2006 midterm election. Adult panelists residing in five states holding elections were selected to participate in the poll and asked to identify the candidate they would vote for in Governor and U.S. Senate races in their state if the election were being held that day. Respondents also answered a series of questions about their past voting behavior and current voting intent to determine who were most likely to vote. We focus our analysis on a subset of likely voters.
Because poll respondents are active Gallup Panel members, additional information is available including the mode (mail or web) that they are assigned for panel surveys. This allows us to use a third mode (phone) to compare the responses of those panelists who are assigned to mail or web and thus, we are able to control for any differences in responses that may be attributable to visual differences in question layout (i.e., mode differences between mail and web). Response rates and the number of likely voters in each state in each comparison group are detailed in table 3. In order for the sample of panelists in each state to reflect the adult population of that state, we applied weights for gender by age, education, race, Hispanic ethnicity, and by region as defined by clusters of counties thought to be meaningfully distinct for electoral purposes. An additional set of web-only weights were computed in order to compare the results with those that would have been obtained if only web panelists were polled.
|
Our decision to restrict our analysis to the voting intentions of only likely voters in the Gallup Panel resulted in the disadvantage of losing a large number of cases. For example, about 41 percent of those polled in State A were not deemed likely voters and were excluded. When we consider the voting intentions of the web-assigned group, the resulting number of cases in each cell is less than optimal. Because statistical power has been lost, we are cautious in our interpretation of results but feel they are important to consider because the approach is methodologically interesting.
|
Results and Findings |
|---|
|
TOPAbstractIntroductionBackground for this ResearchData and MethodsResults and FindingsDiscussion and ConclusionsReferences |
|---|
Do gallup panel respondents differ in behavior and demographic characteristics by assigned mode of response?
Consistent with our expectations and prior research on the demographic differences between Internet users and non-Internet users, panelists assigned to mail had much lower household incomes and lower levels of educational attainment than panelists assigned to the web (see table 1). Web panelists are more likely to be married, employed full-time and younger than mail panelists. Panelists assigned to respond by web are closer to the general public in terms of gender and age than those who respond by mail. Mail-assigned panelists are closer to the general public on income, college degree, and marital status than web-assigned panelists (U.S. Census Bureau 2000
; U.S. Census Bureau/Bureau of Labor Statistics 2005 Current Population Survey; see also Knoef and de Voss 2008). However, it is important to recognize that panelists assigned to both modes report higher incomes, higher education, are more likely to be married, male, non-Hispanic white, and older than the U.S. population.
Do mail and web panelists provide different answers (and item-nonresponse rates) to the same survey questions?
Before evaluating whether mail and web respondents give different answers, we first examined item nonresponses and responses to all 92 items in the World Affairs survey. The mean item-missing rate is 2.5 items per respondent, mail respondents left an average of 3.2 items unanswered, and web respondents left 1.9 items unanswered on average. Based on these results, there does not appear to be substantial differences in nonresponse behavior between mail and web respondents. Thus, it did not seem important to pursue item nonresponse further as a source of differences in mail versus web respondents.
To determine whether or not mail- and web-assigned panelists provide different answers, we compare4 mail versus web responses to attitudinal and behavioral questions separately before and after poststratification weighting adjustments. Before weighting, mail versus web responses differed by five or more percentage points between cells in 30 of the 92 items in the World Affairs survey (see table 4 column c). For example, in answering the question "What do you think foreign trade means for America?" Fifty-three percent of mail respondents answered, "An opportunity for economic growth through increased U.S. exports" in contrast to 67 percent of web panelists—a difference of 14 percentage points. After weighting, mail versus web responses differed in the same magnitude (i.e., at least five percentage points) on 34 of the same 92 items (not shown).
The items that mail and web respondents are most different on (at least five percentage points) appear to be distributed throughout the questionnaire and not concentrated in a particular region or limited to a particular type of question. We found no evidence of a trend in the direction of these differences or a common underlying dimension. Based on these comparisons, it appears that mail and web respondents provide different answers to about one-third of all questions in this particular survey. Moreover, these differences persist after traditional poststratification weighting adjustments.
Does weighting on demographics show that mail is not needed and that the same results could be obtained from a web-only survey?
We performed a common poststratification demographic weighting procedure that, in effect, adjusted for compositional differences between the panel and the U.S. population. As shown in table 4, we compare the weighted web and mail responses (column d) with the separately weighted web responses5 (column e). This provides a practical simulation of how final survey estimates would differ with and without mail respondents.
For 16 of the 92 items,6 weighted web responses were not within five percentage points of the weighted mail and web combined responses. This suggests that the results of a web-only survey would be similar for a majority of questions, but on nearly one-fifth of the items, mail appears to add to the results in a way that cannot be reproduced through traditional weighting of web-only responses.
Does mode of response affect the answers panelists give controlling for demographic characteristics and suggest that mail is not needed?
We use logistic regression7 to gain insight into whether the mode of survey response influences people's answers before and after the available demographic information on respondents is taken into account.8 The results presented in table 5 show that the effect of mode alone is a statistically significant predictor of responses to the questions that mail and web answered most differently. When demographic controls are introduced, the effect of mode remains statistically significant in 27 of the 30 models.
|
Based on these results, it appears that the mode of response is related to the answers respondents give, above and beyond the identifiable demographic differences between respondents. Consistent with the results of the weighting comparisons, this suggests that controlling for a traditional set of demographic characteristics does not eliminate mail versus web differences in the answers given to 27 of the 30 survey questions analyzed here.
Do voting intentions differ between mail-assigned and web-assigned panelists who responded to a preelection telephone poll?
We use data collected from web-assigned and mail-assigned panelists by telephone to conduct a limited test of a mail versus web mode differences. In doing so, we compare the accuracy of election outcome predictions based on whether or not responses from mail-assigned panelists are included, thus removing the potential for answers to differ due to mode of response (mail versus web). This comparison is particularly important because it is the only one that uses a "neutral" mode to collect information from mail- and web-assigned panelists, as well as the only one to have a known behavioral outcome that can be used to examine predictive validity, i.e., the percent of votes obtained by election candidates.
Poststratification weights (age, education, gender, race, and Hispanic ethnicity were applied to the mail–web combined group and web-assigned group separately to simulate how final election predictions would differ with and without the inclusion of mail-assigned respondents. Using responses from mail-assigned and web-assigned likely voters, we calculated the predicted percentage point difference in winner's percent of final vote and margin of victory (which takes loser's percent of the vote into account) and compared our estimates to the final election results. On average, responses from the mail and web group differed from the final results by 1.9 percentage points while the responses from the web-assigned group differed by 3.1 percentage points (table 6). The differences did not consistently favor one party's candidate or another. Substantively, this is an interesting finding but given the smaller sample sizes available for web-only estimates, differences of this magnitude could be expected due to variable error. In light of this limitation, we hesitate to conclude that combined mail and web estimates were truly superior in reducing bias.
|
The predicted margin of victory based on the mail and web combined group also appeared to be more accurate on average than for the web-assigned group. For example, when predictions are based on the mail–web combined group's responses, we would expect the margin of victory in the State A senate race to be 25.6 percentage points when the final results reveal a margin of 24.3 percentage points—a difference of 1.3 percentage points. If responses were only obtained from web-assigned respondents, we would expect the margin to be 25.9 points—a difference of 1.6 percentage points. When averaged across the nine races, responses from the mail–web combined group appear to more accurately reflect the final margin of victory than the web-only group (4.6 versus 5.4 percentage point difference, respectively).
The inclusion of mail-assigned panelists could add to the accuracy of results in a way that is not reproduced when only web-assigned panelists are polled. This limited test is consistent with the idea that mail and web responses do not differ simply because the modes produce different answers. Differences between web-assigned and mail-assigned panelists can be observed regardless of the mode. These results should be interpreted with caution, however, because of the loss of statistical power associated with restricting the analysis to only likely voters rather than all respondents (see table 3).
|
Discussion and Conclusions |
|---|
|
TOPAbstractIntroductionBackground for this ResearchData and MethodsResults and FindingsDiscussion and ConclusionsReferences |
|---|
In this paper, we have used several approaches to examine whether a probability-based household panel benefits from assignment to mail response as alternative to Internet-only. Overall, it appears that use of the mail procedure adds value to the Gallup Panel in the survey examined here. When mail is used as a data collection mode, it is possible to obtain responses from people with quite different demographics than those of the web-only respondents, especially with regard to age, education, and income. Moreover, mail versus web differences in answers to opinion and behavior questions in the World Affairs survey resulted in sizable differences in the survey estimates for a number of questions. When the effects of weighting aimed at simulating survey results with and without mail respondents were examined, it was concluded, as others have found in separate analyses, that traditional poststratification weighting was inadequate. The weighted web results alone did not reproduce final survey results that would be obtained if mail were included as a data collection mode. Mode of response was a statistically significant predictor of answers given, beyond the standard set of demographic characteristics we considered. Finally, when a neutral mode of data collection was used for web and mail panelists, the predictions of an election outcome from a preelection telephone poll might have been improved by combining the mail-assigned and web-assigned results.
While our results suggest that the use of mail questionnaires adds value to the data collection process being used in the Gallup Panel surveys, it is important to note that the overall magnitude of differences between answers provided in the mail and web questionnaires reported here are not large for most of the survey questions. It is also important to recognize that we chose to examine one panel survey in detail rather than consider multiple panel surveys. This research is cross-sectional and we have not examined how using mail questionnaires in the way we have described may influence data quality over time and across several panel surveys because of possible conditioning effects from repeated surveys.
The potential effects of survey mode on responses could not be effectively addressed here because of the nonrandom assignment of households to web versus mail. Considerable research now suggests that differences in how people answer questions in these modes are affected by visual layouts (Dillman, Smyth, and Christian 2009). Because an explicit effort was made to achieve quite similar visual presentations for all questions, and because of results from the test of web versus mail panelists through a neutral (telephone) mode, we doubt that mode is the cause of differences reported here. However, it remains an important issue for further research.
The results of this analysis suggest implications for panel procedures as well as new research. The mail portion of the panel studied here includes a number of people placed there by preference or infrequency of use; 18 percent use email more than once per week but asked to be surveyed by mail and an additional 9 percent were assigned to mail because of being on the Internet only once a week or less. Panel sponsors placed them in the mail portion of the panel because of the expectation that people were more likely to respond if placed there, contrary to preferences or infrequent use, which itself could be an indicator of a lack of familiarity and comfort with being on the Internet. It seems plausible that some of these individuals could be effectively surveyed over the Internet. Efforts to convert such respondents to the Internet and determine whether they will respond as well as frequent users of the Internet seem warranted.
Some Internet-only probability panels attempted to include non-Internet users by giving them computers and/or Internet connections (e.g., Knowledge Networks in the U.S.A. and the LISS Panel in the Netherlands). We know of no published research that has evaluated the extent to which this technique is effective in getting nonusers to respond to such panels and whether the collection of such data contributes to the accuracy of the panel results. Such research is also needed.
For as long as a significant portion of the U.S. population and/or potential respondents are reluctant or lack the skills to respond to questionnaires via the Internet, it seems likely that surveyors may need to consider the use of second or third modes to collect data from those individuals. The panel data analyzed here represents one way of attempting to supplement Internet with mail response. Because those additional responses come from respondents who are quite different demographically from the web respondents, including them appears to contribute to the accuracy of results. Future research needs to study the effectiveness of other ways of using supplemental modes as well as whether giving people Internet access, and convincing them to use it, improves the accuracy of survey results.
An additional problem faced by probability-based Internet panels is reliance on RDD to select household respondents. Not only is the percent of households with landlines continuing to decline (Link et al. 2008), but the scientific adequacy of current sampling methods is also being questioned (Fahimi, Kulp, and Brick 2008). At the same time, it now appears that U.S. residential households can be contacted through use of the U.S. Postal Service Delivery Sequence File and that coverage may exceed that available through RDD (Link et al. 2008). Preliminary experimentation using this address-based sampling in a cross-sectional regional survey has shown that a significant proportion of households contacted by mail can be convinced to respond via the Internet (41 percent) and other respondents (14 percent) by mail (Dillman, Smyth, and Christian 2009, p. 235). The demographics of these mail respondents differ significantly from those of web respondents. However, it remains to be seen whether such methods offer promise for Internet panel surveys on state or national populations of households.
Future research on general public Internet panels needs to address alternatives to RDD, just as it needs to address whether Internet-only response procedures are adequate. Although a great deal has been learned about alternative ways of building general public Internet panels, much remains to be learned before conclusions can be made about their effectiveness for obtaining high-quality data from the general public.
|
Footnotes |
|---|
BRYAN D. ROOKEY AND DON A. DILLMAN are with Social and Economic Sciences Research Center, Washington State University, P.O. Box 644014, Pullman, WA 99164-4014, USA. STEVE HANWAY is with The Gallup Organization, 901 F Street NW, Suite 400, Washington, DC, USA. We gratefully acknowledge with thanks help with earlier versions of the paper provided by Julie Curd, Darby Miller Steiger, Larry Curd, and Ron Bartholomew of The Gallup Organization; Zac Arens, formerly of The Gallup Organization; and Jolene Smyth at the University of Nebraska-Lincoln. Analysis of these data was supported by funds provided by The Gallup Organization to the Social and Economic Sciences Research Center at Washington State University.
1 This decision was reinforced by panel managers who indicated that whereas requests were frequently received from panel members to switch from Internet to mail, it was quite rare for anyone to ask to switch from paper to web. 
2 The questionnaire is available from authors upon request.
3 Household income is not included due to high number of cases for which information is unavailable (n = 4,890).
4 Given the large number of cases under consideration (over 13,000 cases for each mode), even small mail versus web differences are statistically significant. Because of this, we elected not to report results of individual significance tests but note that differences in mail and web responses are statistically significant for 87 of the 92 questionnaire items in the World Affairs. Our use of five percentage points as a benchmark of substantial differences in mail versus web responses is somewhat arbitrary. We contend that a difference of five percentage points is relatively substantial considering the large number of cases in the comparisons.
5 We created two sets of poststratification weights: (1) mail and web combined, and (2) web-only weights to match what would be used for a survey of only web respondents.
6 Of the 16 differences, 8 are displayed in table 4, column f. The other eight items were ones not initially at least five percentage points different between modes before web-only weights were applied.
7 The full model being estimated is (Pr
Yi = 1) = β0 + β1PAPER + β2DEGREE + β3MARRIED + β4FULLTIME + β5RESSTAB + β6AGE + β7MALE + β8SOUTH + β9NUMCHILD + β10NHWHITE + β11RURAL + 
. For ease of interpretation, we report odds ratios (see Long 1997; Pampel 2000)
8 Based on variance inflation factors (VIF) and correlations between independent variables, we feel confident that our models do not suffer from serious and consequential problems of collinearity among independent variables.
|
References |
|---|
|
TOPAbstractIntroductionBackground for this ResearchData and MethodsResults and FindingsDiscussion and ConclusionsReferences |
|---|
Bethlehem Jelke, Stoop Ineke. Online Panels—A Paradigm Theft? (2007) Paper presented at the Annual Meeting of the Association for Survey Computing, Southampton, England. Retrieved May 20, 2006 from www2.asc.org.uk/ASC/Sep2007/Conference/files/papers/asc13-Jelke%20Bethlehem.pdf.
Biemer Paul P., Christ Sharon L. Weighting Survey Data. In: International Handbook of Survey Methodology—de Leeuw Edith, J. Hox Joop, Dillman Don, A., eds. (2008) New York: Taylor & Francis Group.
Blumberg Stephen J., Luke Julian V. Wireless Substitution: Early Release of Estimates Based on Data from the National Health Interview Survey, July-December 2006. (2007) National Center for Health Statistics. Retrieved November 6, 2007, from http://cdc.gov/nchs/nhis.htm.
Council of American Survey Research Organizations. Code of Standards and Ethics for Survey Research. (2008) Retrieved May 21, 2008, from http://www.casro.org/codeofstandards.cfm.
Couper Mick P. Web Surveys: A Review of Issues and Approaches. Public Opinion Quarterly (2000) 64:464–94.[CrossRef][Web of Science][Medline]
Couper Mick P., Kapteyn Arie, Schonlau Matthias, Winter Joachim. Noncoverage and Nonresponse in an Internet Survey. Social Science Research (2007) 36:131–48.[CrossRef][Web of Science]
Dillman Don A., Smyth Jolene D., Christian Leah Melani. Internet, Mail and Mixed-Mode Surveys (2009) Hoboken, NJ: Wiley.
Duffy Bobby, Smith Kate, Terhanian George, Bremer John. Comparing Data from Online and Face-to-Face Surveys. International Journal of Market Research (2005) 47:615–39.
Fahimi Mansour, Kulp Dale, Brick J. Michael. Bias in List-Assisted 100-Series RDD Sampling. In: Survey Practice (2008) September 2008. Retrieved October 20, 2008 from http://surveypractice.org/2008/09/25/bias-in-list-assisted-100-series-rdd-sampling/.
Kehoe Colleen M., Pitkow James E. Surveying the Territory: GVU's Five WWW User Surveys. The World Wide Web Journal (1996) 1:77–84.
Knoef Marike, deVoss Klass. The Representativeness of the LISS Panel. (2008) Paper presented at the MESS Workshop, August 23, 2008, Zeist, The Netherlands.
Lee Sunghee. Propensity Score Adjustment as a Weighting Scheme for Volunteer Panel Web Surveys. Journal of Official Statistics (2006a) 22:329–49.
Lee Sunghee. An Evaluation of Nonresponse and Coverage Errors in a Prerecruited Probability Web Panel Survey. Social Science Computer Review (2006b) 24:460–75.
Link Michael W., Mokdad Ali. Can Web and Mail Survey Modes Improve Participation in an RDD-Based National Health Surveillance? Journal of Official Statistics (2006) 22:293–312.
Link Michael W., Battaglia Michael P., Frankel Martin R., Osborn Larry, Mokdad Ali H. A Comparison of Address-Based Sampling (ABS) versus Random-Digit Dialing (RDD) for General Population Surveys. Public Opinion Quarterly (2008) 72:6–27.
Long J. Scott. Regression Models for Categorical and Limited Dependent Variables (1997) Thousand Oaks, CA: Sage.
McDevitt Paul K., Small Michael H. Proprietary Market Research: Are Online Panels Appropriate? Marketing Intelligence and Planning (2002) 20:285–96.[CrossRef]
National Telecommunications and Information Administration. A Nation Online: Entering the Broadband Age. (2004) U.S. Department of Commerce, Washington, D.C. Retrieved May 28, 2008, from http://www.ntia.doc.gov/reports/anol/NationOnlineBroadband04.pdfA-1-A-2.
Pampel Fred C. Logistic Regression: A Primer (2000) Thousand Oaks, CA: Sage.
Pew Internet and American Life Project. February 15-March 7, 2007 Tracking Survey. (2007) Retrieved September 26, 2007, from www.pewinternet.org/trends/User_Demo_6.15.07.htm.
Sparrow Nick, Curtice John. Measuring the Attitudes of the General Public via Internet Polls: An Evaluation. International Journal of Market Research (2004) 46:23–45.
Terhanian George. How to Produce Credible, Trustworthy Information Through Internet-Based Survey Research. (2000) Paper presented at the Annual Conference of the American Association for Public Opinion Research, Portland, OR, USA.
U.S. Census Bureau. Current Population Survey, Annual Social and Economic Supplement. (2005) (February 2007) http://www.census.gov/cps.
U.S. Census Bureau. Census 2000. Summary File 1, 3; Generated Using American FactFinder. http://factfinder.census.gov (February 2007).
Vehovar Vasja, Manfreda Katja Lozar. Web Surveys: Can Weighting Solve the Problem? (1999) 962–67. Proceedings of the Section on Survey Research Methods, American Statistical Association.
Witte James C., Amoroso Lisa M., Howard Philip E. Method and Representation in Internet-Based Survey Tool—Mobility, Community and Cultural Identity in Survey 2000. Social Science Computer Review (2000) 18:179–95.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||