Public Opinion Quarterly Advance Access originally published online on January 29, 2009
Public Opinion Quarterly 2008 72(5):1008-1032; doi:10.1093/poq/nfn065
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
This article appears in the following Public Opinion Quarterly issue: Special Issue: Web Survey Methods [View the issue table of contents]
Computing Response Metrics for Online Panels
Address correspondence to Mario Callegaro; e-mail: mcallegaro{at}knowledgenetworks.com; mail at Knowledge Networks, Inc., 1350 Willow Road, Ste. 102, Menlo Park, CA 94025.
 |
Abstract |
|---|
TOP
Abstract
Types of Online Panels...As more researchers use online panels for studies, the need for standardized rates to evaluate these studies becomes paramount. There are currently many different ways and conflicting terminology used to compute various metrics for online panels. This paper discusses the sparse literature on how to compute response, refusal, and other rates and proposes a set of formulas and a standardized terminology that can be used to calculate and interpret these metrics for online panel studies. A description of and distinction between probability-based and volunteer opt-in panels is made since not all metrics apply to both types. A review of the existing discussion and recommendations, mostly from international organizations, is presented for background and context. In order to propose response and other metrics, the different stages involved in building an online panel are delineated. Metrics associated with these stages contribute to cumulative response rate formulas that can be used to evaluate studies using online probability-based panels. (Only completion rates can be calculated with opt-in panels.) We conclude with a discussion of the meaning of the different metrics proposed and what we think should be reported for which type of panel.
Online panels are increasingly being used as a mode of data collection for market (Postoaca 2006a
; Comley 2007), social (Tortora 2009), psychological (Göritz 2007), and medical research (Couper 2007). Market research is the sector relying most heavily on online panels. Comley (2007), citing data from Inside Research, reports that in 2006 one-third of all market research in the United States was conducted online with a similar trend in Europe. Because of the relative newness of online panels, no standardized way has emerged to compute response metrics. The terminology used by different companies and institutions varies and often the same term is used with a different meaning. The goal of this paper is to attempt to conceptualize response metrics in the realm of online panels, organize the sparse literature on the subject, and propose new metrics. For context, we present a review of the steps in creating an online panel and how each of these steps has an impact on response rates. We hope to provide a framework to discuss and report response metrics in the literature of online panels and offer a common language so that readers can clearly understand what is being reported.
|
Types of Online Panels and Metrics being Proposed in the Literature |
|---|
Taxonomy of online panels
On applying Couper's (2000
) taxonomy on types of Web surveys to online panels, online Web panels can be classified into prerecruited probability-based Web panels or volunteer opt-in panels (see also Lozar Manfreda and Vehovar 2008; Sikkel and Hoogendoorn 2008). Prerecruited probability Web panels are composed of respondents who have been selected with some kind of probability method, for example, panel members being recruited through random-digit dial (RDD) telephone sampling. The recruitment can also be done by email, mail or face-to-face, as long as there is a known nonzero probability of selection from a given sampling frame. Knowledge of the sampling frame and the recruitment methodology enables the researcher to measure coverage and nonresponse error which can be used to properly weight and adjust the recruited participant data.
Volunteer panels of Web users (Couper 2000) or opt-in panels are composed of respondents who voluntarily sign up (opt-in) to become members of the panel. In the European literature these are frequently referred to as access panels (European Federation of Associations of Market Research Organizations 2004). These respondents might have found out about the panel via word of mouth, any kind of advertisement or referral, or been recruited via pop-ups or other methods of discovery. Recently, some Websites have been specifically designed to "sign up" respondents to several opt-in panels at once. Examples are surveymonster.net and yellowsurveys.com.
Recruitment methods for online panels (probability-based or opt-in) have evolved during the years, and they have an impact on the meaning of both response metrics and nonresponse bias. For example, it is becoming popular to recruit panel members at the end of an offline survey (Comley 2007). In that case, the panel generated is a subset of the respondents to the initial offline survey. For an in-depth description of recruitment methods for online panels, the reader is referred to Miller (2006) and Comley (2007).
Because the opt-in panel frame is unknown, it is not possible to fully compute response rates for these volunteers as there is no denominator (Fricker and Schonlau 2002; Schonlau, Fricker, and Elliott 2002; American Association for Public Opinion Research 2006). However, we want to propose some possible metrics that can be used in the industry because standardized metrics are essential for any kind of comparative or performance evaluation. In order to introduce fewer new terms and keep consistent with the current literature, we will mostly follow the notation and definitions published in the AAPOR Standard Definitions (American Association for Public Opinion Research 2006).
Previous literature on metrics for online panels
Discussion of possible metrics for online panels can be found in two sources: national and international professional associations, and scholarly research. For an historical perspective on response standards, the reader is referred to Smith (2002). We start our review from professional associations’ guidelines.
Several professional associations have released guidelines for computing metrics for online panels. The first instance can be found in a consortium of German market and social research institutes that released their "standards for quality assurance for online surveys" as early as 2001 (Arbeitskreis Deutscher Markt 2001). The European Federation of Associations of Market Research Organizations (EFAMRO) developed a document on "Quality standards for access panels" (European Federation of Associations of Market Research Organizations 2004), substantially based on the European Society for Opinion and Marketing Research (ESOMAR) guidelines for online surveys. The document suggests a minimum standard for measuring and reporting nonresponse and completion rates. ESOMAR issued a document on "Conducting market and opinion research using the Internet" (European Society for Opinion and Marketing Research 2005) and more recently "26 Questions to help research buyers of online samples" (European Society for Opinion and Marketing Research 2008). Lastly, the Interactive Market Research Organization (IMRO) released a document called "Guidelines for best practices in online sample and panel management" (Interactive Marketing Research Organization 2006). The terminology in these reports is used interchangeably and in only a few instances is there a clear definition of some response metrics (e.g., ESOMAR, IMRO).
In the ESOMAR (2005) document, response rates are discussed in the section devoted to project management and in the project technical summary. In the summary, response rates should be reported as follows:
"Response based on the total amount of invites (% of full numbers) per sample drawn (country, questionnaire); % questionnaire opened; % questionnaire completed (including screen-out); % in target group (based on quotas); % validated (the balance is cleaned out, if applicable)" (p. 20). Later on in the document, ESOMAR suggests 25 questions to help research buyers, question 16 being: "What are likely response rates and how is response rate calculated?"
IMRO (2006
In the scholarly research, the first mention of panel metrics comes from the literature on computer-assisted panel research (Saris 1991). In a later paper describing the prerecruited probability-based Dutch Telepanel, Saris (1998) explains the initial response rate as the percentage of people who initially agree (on the phone) to be members of the panel and complete a second in-house interview. Later on, Sikkel and Hoogendoorn (2008) introduced response rates for each of the different stages of recruitment in the Dutch CentERpanel.
A term that is now becoming widely used in the literature on prerecruited probability Web panels is the cumulative or multiplicative response rate (Huggins and Eyerman 2001; Schlengen et al. 2002; Tourangeau 2003; Office of Management and Budget (OMB) 2006; Schonlau et al. 2006; Couper 2007). The cumulative response rate is computed by multiplying the response rate for three stages of panel buildup: recruitment, connection, and specific-study response.
Some journals advocate the use of the AAPOR Standard Definitions (2006)1 when submitting papers involving Web surveys (e.g., Public Opinion Quarterly, International Journal of Public Opinion Research), while others such as the Journal of Medical Internet Research give specific guidelines for reporting response rates in online studies. It actually suggests avoiding the term "response rate" and instead suggests using the journal's definitions of "view rate," "participation rate," and "completion rate" (Eysenbach 2004).
|
Recruitment Strategies in a Prerecruited Probability-Based Web Panel |
|---|
Understanding the process of how a panel is recruited and put together provides the basis for response metrics. Generally speaking, prerecruited probability-based panels implement the following methodology to recruit a panel member. The first step is to contact and recruit the potential members via telephone, face-to-face, mail, or email. The next step is to administer a "profile" or "welcome" survey in order to collect basic demographic information on the potential panelists. Different survey organizations have various ways of collecting such information, but there is at least one survey or data collection step to be completed in order for the recruited respondents to become part of what can be termed as the "active panel." The active panel is the pool of now available members that can be sampled and assigned to a specific survey. Figure 1 exemplifies a typical Web panel recruitment process.
|
Because time enters the picture, some disposition codes that are established at one point in time in a traditional cross-sectional study are determined over several points in time in Web panel recruitment. At each stage some disposition codes are not applicable, thus simplifying the picture. In figure 1 we can see these concepts in practice. During the initial recruitment by a survey organization, we will have cases of known or unknown eligibility. The known eligible cases will then be classified into two groups: eligible and not eligible. The survey organization will try to recruit as many people as possible among the eligible cases. These respondents will be asked to become members of the panel. At a second point in time, a profile or demographic survey will be administered to the "initial consent" respondents. Because these people have been determined to be eligible, the only outcomes possible in the profile survey are a returned questionnaire (i.e., either complete or partial), and an eligible noninterview. We have then three outcome classes: (1) refusals and break-offs, (2) noncontacts, and (3) other noninterviews.
Profiled recruits become members of the active panel. Some or all of them will eventually be part of a sample assigned to a specific survey. The sample selection can be random, can be based on specific qualifying criteria available from the profile interview, can result from screening questions asked from sampled panelists, or all active panel members can be selected and assigned. Once a sample selection is made, an email invitation is sent to the assigned members with a link to the study's survey. Possible outcomes from this email invitation step are shown in figure 2.
|
Figures 1 and 2 depict ideal cases, although there might be variations in the number of steps necessary to draw and assign the sample for a specific study. The key concept in computing response rates for a probability-based panel is the consideration of one or more steps in the process before computing the necessary metrics for the study in question.
description of data collection designs for some prerecruited, probability-based web panels
Examples of prerecruited, probability-based panels are the pioneer (now discontinued) Dutch Telepanel, the current CentERdata CentERpanel and the Long-term Internet study for the Social Science (LISS) in the Netherlands. For the United States, examples are KnowledgePanel® and the Gallup PanelTM. At the time of this writing, more probability-based panels were in the process of being built and evaluated.
The Dutch Telepanel was a prerecruited, probability-based computer-assisted panel that originated in the 1980s. The recruitment telephone interview obtained basic demographic and other background data. The household members who agreed to become members of the panel were then visited by an interviewer who brought a computer to the respondent home, installed it, and taught eligible panel members how to use it (Saris 1998). During that visit, the interviewer collected more background information on the household, thus fulfilling the goal of an initial profile survey. All household members were invited to join the panel, including children of reading age.
The CentERdata CentERpanel recruits households via telephone. The household members are asked if they want to participate in survey research projects. If so, they are included in a database of potential panel members and some of them are then selected to become part of the panel. All members of the chosen household are invited to join. For households that do not have a PC or Internet connection, a set-top box is provided and plugged into the TV, thus enabling Internet access (Toepoel, Das, and van Soest in press; Hoogendoorn and Daalmans in press). LISS is a new representative panel of 5,000 Dutch households built similarly to the CentERpanel with an emphasis for longitudinal and academic research. Respondents 16 years and older are asked to participate.
The Knowledge Networks KnowledgePanel data collection process follows approximately the stages described in figure 1. In the recruitment stage, conducted by RDD, Knowledge Networks tries to recruit every member of the household 13 years of age and older. This is done by asking the informant to complete a household roster where basic demographic information is collected for each member 13 years and older. After people are recruited, the potential households are classified into two groups—those who have Internet access and those who do not. Knowledge Networks provides a WebTV to the non-Internet households, or more recently a laptop computer is provided. The WebTV, via a telephone modem, transforms a television into a Web browser. Additionally, all members receive a welcome survey that educates them on how to navigate and fill out Web surveys. The next step involves the completion of a profile questionnaire to collect basic demographic information on respondents and their households (Huggins and Eyerman 2001; Pineau, Nukulkij, and Tang 2006). After the profile questionnaire is completed, respondents enter the pool of active panel members.
The Gallup Organization recruits respondents for the Gallup Panel via RDD. Households willing to participate are sent a "membership packet" by mail with a demographic questionnaire to be completed and mailed back. The company tries to recruit all members of the household aged 13 years and older. Household members returning a completed membership packet become part of the active panel (Arens and Miller Steiger 2006; Sayles and Arens 2007; Tortora 2009). Although the Gallup Panel collects survey data using different modes (Web, mail, telephone, IVR), the formulas proposed in this paper apply, with the difference that some disposition codes have to be adapted for the applicable mode of data collection. The stages of recruitment coincide with the process depicted in figure 1.
In 2008, Stanford University recruited a national area probability sample of adults and equipped them all with a free laptop computer and high-speed Internet access; the project tested the feasibility of recruiting such a panel to complete monthly surveys for a year. Also in the same year, the American National Election Study (run by the University of Michigan and Stanford University) recruited a Web-enabled panel via RDD providing non-Internet households with a WebTV.2
Lastly, the American Life Panel maintained by RAND consists of approximately 1500 respondents recruited by telephone from the University of Michigan Survey of Consumer Attitudes. Non-Internet households are provided with a WebTV3 (see also Couper and Dominitz 2007).
More probability-based panels are in the process of being built and/or evaluated. For example, an evaluation of an initial mail recruitment strategy for the CentERdata panel in the Netherlands is going on (Vis 2007).4
|
Enrollment Procedures for Volunteer Opt-In Panels |
|---|
According to Postoaca (2006a
), volunteer opt-in panels can be classified into two categories defined by their enrollment procedures: single opt-in enrollment and double opt-in enrollment. Figure 3 provides a schema of these procedures.
|
The potential opt-in panel members start by either going to the specific panel recruitment portal, by being redirected through banners, or by using one of the opt-in panel consolidators’ pages mentioned before. The potential respondents enter some basic information about themselves including an email address at the panel's recruitment portal. If it is a double opt-in enrollment, they receive an email confirmation with a link that they have to click to get to the enrollment page. If it is a single opt-in process, the email confirmation is omitted and they are sent directly to a "recruitment questionnaire" page (Postoaca 2006a
) where more demographic/profile information is collected. Next, whether single or double opt-in, they become active panel members ready to be selected and assigned to a study. There are some variations in steps among volunteer opt-in panels. Some panels can have members immediately assigned surveys before completing their profile questionnaire. In this case, the demographic questions typically collected in a unique profile questionnaire are instead collected at the end of an introductory survey or in parts among different surveys.
There is not enough space to present all of the recruiting variations but single and double opt-in are the two major defining elements. Double opt-in appears to be emerging as the best practice (Miller 2006; Comley 2007). Comley (2007) suggests using the double opt-in procedure, recruiting from a variety of sources, to minimize the chance of people enrolling more than once. Additionally, he suggests administering a detailed "enrollment registration survey" in order to have adequate information to assess eligibility for specific surveys.
|
The Concept of an "Active Panel" |
|---|
Following recruitment, the concept of an active panel applies to both probability-based and volunteer opt-in panels, although with different implications. All online panels are generally very dynamic with members joining and leaving. Unlike longitudinal household panels, recruitment is an ongoing activity. Figure 4 represents the dynamics of an active panel.
|
The size of an active panel varies over time and depends on different operational actions. Continuous recruitment provides an inflow of potential panel members, some of them eventually ‘connecting’ and becoming active members. Simultaneously, at any point in time there are members leaving the panel voluntarily or involuntarily. In the latter case, online panels have different rules of "purging" the database of members who did not respond consecutively for x number of surveys. Panel management can really make a difference, and some companies are more aggressive than others in retaining members and keeping them participating in surveys. In some instances panels have rules about maximum tenure, i.e., panel members are forced to leave after a specific amount of time.
Given the cost of recruitment, especially for probability-based panels, it can be more cost effective to rerecruit former panel members who have voluntarily left. Rerecruitment initiatives using incentive strategies may be launched to recover former panel members. At any given point in time, some active members may become temporarily inactive due to a variety of reasons. Members might voluntarily communicate periods of time during which they are not available due to vacation, illness and the like. On the other hand, the survey organization might make some members "not available for sampling" due to maximum burden rules—completing n surveys in a given time period. Many online panels have such rules to avoid overburdening panel members.
Another reason for not being available for a particular study sample is specific requests of the study in question. A client may not want sampled members to have completed a recent survey on a similar topic or on a topic that might influence the survey outcome. This is similar to screening but may be more akin to a quarantine-like criterion (Postoaca 2006a). These above-mentioned cases can all be classified as ineligible for a particular study.
The concept of an "active panel" has repercussions when computing metrics because of the availability of different panel members at a given point in time. By aggressively retaining, say, the most cooperative panel members, it is possible to greatly increase the survey completion rate, as shown by Vonk, van Osenbruggen, and Willems (2006). The authors sent the same survey to 19 online opt-in panels in the Netherlands exactly at the same time. The survey completion rate had a range from 18 to 77 percent. The authors conclude: "Response percentage does not indicate sample or panel quality. It reflects a panel business strategy. The response rate is an indication of the level of efficiency of the panel provider" (p. 20). Although keeping the most cooperative panel members can be seen as a winning characteristic for some volunteer opt-in panels, for probability-based panels it is a problem because it has an effect on representativeness of the panel as well as on the cumulative response rates.
|
Computation of Response Metrics |
|---|
At the moment there is no standard to compute response metrics for Web panels. There are, however, standards on how to compute response rates for Web surveys. In 2006, AAPOR issued the fourth version of the Standard Definitions document (American Association for Public Opinion Research 2006
) that included "Internet surveys of specific named persons" which details how to compute response rates for Web surveys. As described earlier, there are many stages involved in building online panels so we propose computing a rate for each stage. We present formulas with the maximum operands, so the reader can derive other formulas by dropping some operands.
Stage 1 recruitment rate
Stage 1 applies to probability-based Web panels because for volunteer opt-in panels the base is unknown. Depending on the panel design, all eligible household members can be invited to join the panel or a within-household selection method can be employed that selects one eligible member only. At Stage 1, a recruitment rate (RECR) can be computed as follows:
|
|
where
- IC = initial consent
- R = cases directly and actively refusing
- NC = noncontacts
- O = other cases
- UH = unknown if household is occupied
- UO = unknown other
- e = estimated proportion of cases of unknown eligibility that are eligible.
- R = cases directly and actively refusing
For the computation, we are using a formula similar to AAPOR RR3. This is the most complex formula; researchers can use a simplified version where UH and UO are not taken into account and also where e is not estimated. The specific disposition codes used depend on the data collection method used, i.e., telephone, face-to-face, mail, or email. See the AAPOR Standard Definitions for the various disposition codes for these survey modes.
If members are recruited during the first contact with the household, the recruitment survey and the agreement to join the panel (IC) are combined into one rate (Couper et al. 2007). If there are steps that take place at two different points in time as exemplified by Arens and Miller Steiger (2006), then RECR represents only the second step, consent to join the panel.
When all eligible members of the household are recruited, RECR can be computed at either the household or the person level. For a household-level application, the terms in the denominator describe the total number of eligible households. In order for the household to be counted in the denominator, each must have at least one potentially eligible member to be recruited.
When there is a within-household selection at the recruitment stage so that only one member per household is recruited for the panel, then RECR computed at a household level is the same as RECR computed at the person level. In the case where multiple members per household have been recruited, but a member-level sample for a given study is drawn in which only one random member per household is selected (among all eligible members if there is an eligibility criterion) and no substitutions are allowed, a household-level RECR measure can be used since it is similar to recruiting only one person per household.
When computing RECR at the person level, the total number of eligible persons across all households needs to be known. The denominator will then describe all eligible persons and the numerator is all recruited persons. The factor e would then be a multiplier that gives the estimated number of eligible persons expected from the number of "unknown" or "other" households.
One issue is when there is a break-off during the recruitment interview and information about the total number of eligible persons per household is not collected. In this instance, as well as for the refusals, e is estimated by using the completed cases to compute the total number of eligible persons per household. The case where all eligible persons are recruited poses the issue of proxy consent. Generally, a telephone recruiter speaks with only one member of the household who initially consents for the other persons. Contact with other adults volunteered by the household respondent has to be arranged so that they can individually decide whether to join (note that when a minor is recruited, consent has to be obtained from his or her parent or legal guardian).
Stage 2 profile/connection stage
Stage 2 applies to probability-based and opt-in panels, although its meaning is different for the opt-in type. The reason for this difference is that the measurement of refusals and noncontacts is not as straightforward as it is for probability-based panels. Essentially, this information is unknown for the "single opt-in" panel; however, when a double opt-in procedure is employed (figure 3), then the profile/connection rate may be thought of as the number of people who confirmed at their second opt-in opportunity over the total number who initially opted in. On the other hand, for probability-based panels, an initial profile survey is sent to all those who agreed to become members at the time of recruitment. Alternatively, the survey organization can redirect its recruited respondents to an online registration page that functions as the profile survey. After recruited respondents answer this profile survey, they become part of the active panel (Lee 2006). As we can imagine, some initial probability-based recruits will choose not to complete their profile. These people drop out before being recognized as panel members. To account for all this, we compute a rate corresponding with Stage 2 that we call the profile rate (PROR). In this second stage, there are no unknown eligibility or ineligible cases because these were screened out earlier.
Generally, a profile rate is computed as follows:
|
|
- I = profile survey complete
- P = profile survey partial.
- P = profile survey partial.
This formula is basically AAPOR RR6. The difference is that we focus only on the profile survey. As with RECR, PROR can be computed at either the household level or at the person level. If the sample for a given study is constrained to selecting only one random panel member per household, the household- and person-level PRORs are the same.
Obviously, the people who do not answer a profile survey are lost to active panel membership and thus not accessible for specific surveys. As for any self-administered survey, there will be cases directly and actively refusing (R), and other cases (O). Some cases may involve failure to install computer equipment needed for prospective respondents. Other cases will be noncontacts (NC); for a Web survey, this means no reply from the respondent (Couper et al. 2007) to the profile survey request. Because these people were contacted at the recruitment stage, some special effort can be undertaken to convert them. They can be recontacted to understand their reasons for not completing their profile survey or, if relevant, to confirm that the email invitation for the profile survey actually reached them. Likely, passive refusal behavior counted among the NC adds to the initial refusal rate and should be monitored in search of possible patterns contributing to membership bias. Again, the specific disposition codes will depend on the method through which the profile survey is administered.
Stage 3 specific study considered
Stage 3 applies to both opt-in and probability panels. Here is where active panel members are asked to respond to particular surveys. Let us briefly review the four participation steps within this stage as delineated by Pratesi et al. (2004): an email invitation sent to assigned panel members; access to the Web survey's opening page, clicking "start" to begin, and ultimately completion of the survey questions. At each of these steps nonresponse enters the picture.
Regarding the first step, the concept of "absorption rate" initially introduced by Lozar Manfreda and Vehovar (2002) is useful for discussing panel communications and management. The absorption rate measures the quality of the email list of active members. For example, if every assigned member actually gets the email sent to him or her, these emails are described as 100 percent "absorbed" and would have a rate of 1.0. The absorption rate can be measured as
|
|
- EI = email invitations sent
- BB = bounce back of undeliverable email invitations
- NET = network error-undeliverable emails.
- BB = bounce back of undeliverable email invitations
The absorption rate is an incomplete indicator of how many panel members receive and thus potentially read the email invitation (EI), for example "bounce back" invitations may actually be received. This could have sample size consequences when setting the number of members to receive initial emails in future surveys. The number of invitations might be adjusted upward to compensate for an expected absorption of less than 1. Nonabsorbed email can be due to bounce back (BB), caused by either a wrong email address, a full mailbox, or a network error (NET).
A special and more recent situation is the case when an email invitation ends up in a spam filter and moved to a spam folder or deleted right away. In this case, the survey organization does not always receive any feedback that that email was flagged as spam. For this reason the email appears "absorbed" but practically the respondent has less of a chance to read it. Because there is no feedback that an email has been flagged as spam it is difficult to estimate the size of the problem. Survey organizations can use some tools to test their email invitations and gather a measure of likelihood of spam.
Once invitations with bad addresses have been accounted for as completely as possible, the most intuitive response metric is the survey's completion rate. It is also the one metric most often mislabeled as a response rate. The completion rate is the proportion of those who completed the Web survey among all the eligible panel members who were invited to take the survey:
|
|
The formula is again AAPOR RR6, but just focusing on the specific-study sample. The completion rate (COMR) is a metric that can be directly applied to volunteer opt-in panels. Note that partial interviews (P) are included in the numerator of "completes" when a "partial" condition is allowed in a study. There are published definitions of a partial interview to distinguish it from a complete or a break-off (AAPOR 2006, p. 11). Researchers are "required" to report the definition they use when partial interviews are allowed.
The sampling criterion relative to members and households is important for the completion rate. In many cases, only one member per household is selected for a study to avoid a within-household clustering effect. This sampling criterion (a constraint of one sample person per household) will determine which level of information will be used when computing the cumulative response rate (see the discussion below).
When an incomplete survey does not qualify as an acceptable partial case (if there is a partial survey definition in place), it can be classified as a "break-off." Basically, the survey was opened but not finished. It is possible to calculate a break-off rate for a given Web survey. The break-off rate (BOR) can be computed as follows:
|
|
Screening
A screening activity in any kind of Web panel has its own metric, as well as one to assess an eligibility rate. Screening is often used to find a specific or rare population eligible for a study. It can be done in several ways. One way is to select respondents based on previously collected information recorded as part of each panel member's profile. For example, information collected during the initial demographic profile survey or from some topic-specific profile survey (like health behaviors) can be used to find eligible panel members. This method is highly efficient if all the right data are available. On the other hand, a panel census can be conducted to locate eligible members. This approach can be costly and less efficient since time and nonresponse are issues. Another screening method would be to send email invitations to a "large enough" sample of the panel in which the first few questions function to assess eligibility and terminate those who are not eligible. The sample size would be a function of expected eligibility incidence.
As discussed by Ezzati-Rice et al. (2000), a screening completion rate (S_COMP) and a study-specific eligibility rate (S_ELIG) are computed in a similar way. The first is calculated as follows:
|
|
- SCQ = screening completed and qualified
- SCNQ = number completing the screening questionnaire and not qualified (screened out)
- INV = number of survey invitations sent.
- SCNQ = number completing the screening questionnaire and not qualified (screened out)
SCQ is the number of people who were successfully screened and found to be qualified for the study. The problem with a screening rate is that nonresponse is confounded with the screening. In fact, we do not know if a person qualifies unless they provide that information by answering the screening questions. For this reason, we talk about screening completion rate and not screening rate. The study-specific eligibility rate (S_ELIG) is conceptually similar to an incidence rate and is calculated as follows:
|
|
- SCQ = creening completed and qualified
- SCNQ = number completing the screening questionnaire and not qualified (screened out).
- SCNQ = number completing the screening questionnaire and not qualified (screened out).
Cumulative response rates
Cumulative response rates are applicable to only probability-based panels. The computation takes into account all the stages that go into a panel sample for a specific study, from panel recruitment to study response. It is the product of each of the component rates. The least number of components make up Cumulative Response Rate 1 (CUMRR1):
|
|
The introduction of a fourth component, a retention rate (RETR), would make up Cumulative Response Rate 2 (CUMRR2). For a given recruitment sample from which we get a cohort of active panel members, some proportion of that original cohort remains on the active panel at the time when the study sample is drawn. This proportion is the retention rate. Calculating each of the component rates for the same cohort and further multiplying them by the retention rate gives us CUMRR2 as follows:
|
|
|
Attrition Rates |
|---|
Attrition rates have a completely different meaning for probability-based and volunteer opt-in panels. For probability-based panels attrition is definitely a cost. Attrition has an effect on cumulative response rates and, even more importantly, on the overall representativeness of the panel demographics. If some possibly rare subgroups have higher attrition rates than other subgroups, members of these rare groups will have a higher chance of being included in multiple general population samples. For opt-in panels, attrition is considered less of a problem in terms of replacement cost and panel efficiency because fewer resources are involved in recruiting panel members.
The concept of panel attrition in online panels is different from the traditional definition of longitudinal surveys. In the latter, attrition is defined as the percentage of panel members that cannot be interviewed from one wave to the next. In online panels, attrition is defined as the percentage of members who drop out of the panel in a defined time period. Attrition is measured after the respondents become "active panel members." In an online panel, a member can actively request to be removed, or they can just stop answering surveys. Different organizations have different mandatory attrition rules. For example, in the Gallup Panel if members do not respondent to six surveys consecutively, they are removed (Sayles and Arens 2007).
When considering a specific cohort resulting from one of the samples drawn during a periodic recruiting effort, attrition is measured by counting how many recruits stay in the panel month after month (Clinton 2001). In formula terms
|
|
- ATTR_Mt = attrition at month t in percentage points
- Cohorta@Timet = the specific cohort considered at time t, generally a specific month
- Cohorta@Timet+1 = the same cohort at time t+1, generally the following month.
- Cohorta@Timet = the specific cohort considered at time t, generally a specific month
Researchers can either use months or weeks as units of time as long as they are specified and consistent. Attrition rates are an indicator of panel retention and can be used to study differential survival rates for subgroups of the population, as shown by DiSogra and colleagues (2007) and Sayles and Arens (2007).
Attrition rates are of special importance if there is a longitudinal design to study change over time across the same subjects. The formula for attrition rates in longitudinal designs can be easily adapted substituting time with wave.
|
Discussion |
|---|
We argue that the term response rate is limited, inconsistently defined, and often abused when reporting metrics for online panels. This view concurs with the opinion of the Journal of Medical Internet Research that this term is best to be avoided when reporting research using online panels. So what response metrics are there to apply to online panel research? This paper proposes several and lays out some terminology to assist in addressing this issue so that some quality dimensions of different online panel studies can be equitably compared.
When doing research with probability-based online panels, we recommend that researchers report several metrics. At a minimum these would be the panel's recruitment rate, the profile rate, the study-specific survey completion rate, and the final cumulative response rate defined as CUMRR1. These rates tell us something about the panel, about the survey completion, and the overall response rate. Naturally, the multiplication of several rates will mathematically produce a smaller and smaller number as more factors are introduced. Compared to the response rates in today's reality of rigorous RDD surveys hovering in the 20–25 percent range, a single-digit cumulative response rate in a Web survey (consider, for example, that four component rates of 0.5 gives a 6.25 percent CUMRR2) will likely appear to be very low.
Volunteer opt-in panels, because of cost and speed, are used for a majority of Web studies, especially in commercial market research (Comley 2007). In the psychology field, an increasing number of studies employing random allocation experiments are now conducted using opt-in panels (Göritz 2007; Reips 2007). The "completion rate" appears to be the single most informative metric to report for a volunteer opt-in panel. The interpretation of this rate may reflect the respondent's interest in the survey and/or the ability of the survey company to maximize cooperation. Panel attrition is also informative because, when it is high, it "could be the result of placing surveys that are too long or poor question design" (European Society for Opinion and Marketing Research 2008).
Absorption rates and break-off rates should also be reported for both probability-based and volunteer panel research. This reporting goes beyond the minimum and tells us much more about the panel and the study. The absorption rate is measuring the ability of the survey company to manage and keep up-to-date their database of email addresses and communications with panel members. The break-off rate is a possible indicator of problems in the design of the questionnaire (e.g., too long, boring...) or struggle with technical problems during the survey administration (e.g., streaming media or animations that may "break" a survey at some point).
Study-specific screening completion rates and eligibility rates measure the incidence of a particular phenomenon among panel members. When these rates are significantly different from an external "gold standard," they may indicate issues of question wording in the screener module or respondents purposively self-selecting themselves for a particular study (e.g., to gain rewards) even if they do not really qualify. The latter is an increasing problem in volunteer opt-in panels and much debate has been devoted to the topic (e.g., sessions dedicated to professional respondents at market research conferences). These rates may also reveal a skew in the panel membership along a particular dimension that may raise concerns regarding bias.
In addition to these rates, we also believe that it is the best practice to report the length of the field period with its start and close dates, the number of reminders sent and their form (email, letter, IVR call, or personal call), and the use of any incentive. This information can help to judge the quality of a study. For example, a very short field period may overrepresent respondents who check their email more frequently and other "early responders."
|
Conclusions |
|---|
In this paper, we discussed the current literature on computing response rates for online panels, and we have proposed some formulas that can be applied to probability-based and volunteer opt-in Web panels. The reader should be careful in interpreting the different metrics, and especially in comparing metrics across panels or between a Web panel and another mode of data collection. Factors such as recruitment methods, eligibility rules, number of survey invitations, and mandatory attrition rules (Postoaca 2006a
, 2006b) are different from company to company and should be reported upfront.
We think that it is more important to study and understand nonresponse bias instead of exclusively pursuing high response rates. As has been discussed by Groves (2006) and Groves and Peytcheva (2008), response rates alone are not a very good indicator of the magnitude of nonresponse bias. We also agree with the idea that high costs to increase response rates and a general trend in decline of response rates should not give way to using nonprobabilistic sampling methods. Additionally, auxiliary data from the sample frame can be used to improve the survey estimates and weighting adjustments. Probability-based Web panels are urged to collect as many auxiliary data as possible in order to improve estimates and compensate for nonresponse. These data may also provide an advantage at each step of building the panel to measure those respondents who fail to become active panel members (Hoogendoorn and Daalmans 2008). Finally, monitoring attrition is also crucial to assessing the representative make-up of any panel, especially since attrition is rarely equal across all demographic subgroups.
|
Appendix A. Example: A Knowledge Networks Project Using a Cumulative Response Rate Computation for an Online KnowledgePanel® Sample |
|---|
Study Title: A Comparison of the Estimates from the 2006 General Social Survey National Priority Items: Online, Telephone, and In-Person Modes of Data Collection
The Knowledge Networks (KN) Web panel is a probability-based panel. By definition, all members of the KN Web panel have a known probability of selection. As a result, it is mathematically possible to calculate a response rate that takes into account all sources of nonresponse. In contrast, opt-in Web panels do not permit the calculation of a response rate since the probabilities of selection are unknown. Consequently, opt-in panels are mathematically capable of computing only the survey completion rate representing the final stage of gaining cooperation of survey research subjects, excluding the nonresponse resulting from panel recruitment, connection, and panel retention. The example of a KN Web panel cumulative response rate, provided herein, should not be confused with the reporting of a single-stage response rate, as in the case of a survey completion rate reported for an opt-in Web panel survey.
The selected example is a survey conducted by Knowledge Networks as part of its own methodological research program. This study is an attempt to contribute to previous research on the subject of data collection mode effects comparing specifically the Internet mode of data collection to telephone-based and in-person data collection (Dennis and Li, 2007; Smith and Dennis, 2008). In this study, we controlled for sample source by having all interviews conducted with pre-recruited panelists from KnowledgePanel.
This KN Web panel survey has one sample randomly split and fielded in two overlapping data collection periods. The sample consists of U.S. adults age 18 and over. All fielded sample cases had one email reminder sent three days after the initial email invitation. No monetary incentive was used in this study.
Field Period 1: start 03/29/2006 end 04/26/2006
Field Period 2: start 04/26/2006 end 05/01/2006
Field Period 3: start 05/01/2006 end 05/15/2006
Initial members assigned: 1,688
Final number of interviews completed: 1,428
Below are the components of the response rate calculation and the calculations themselves.
Household recruitment rate (RECR) = 0.326
Panel recruitment is done using RDD telephone methods. The recruitment rate is computed using the AAPOR Response Rate 3 (RR3) for telephone surveys. If at least one member of the household is recruited, the household as a unit is counted in the household recruitment rate.
Of the 1,688 assigned members in this study, their mean household recruitment rate is 0.326. In the application of the AAPOR RR3 formula to this specific study, the mean numerator across all replicates that were drawn in this sample is 352.4 and the corresponding mean denominator is 1,080.9. It is important to note that when there is continuous recruitment throughout the year for KnowledgePanel, the recruitment rate for any study sample is calculated using the recruitment numbers from the panel recruitment sample's replicate for each study sample member. (For each RDD recruitment sample replicate fielded that donates a case to a given panel study sample, an AAPOR RR3 numerator and denominator is calculated.) To compute the RECR rate for this study, 534 distinct replicates are involved. The relevant replicate's recruitment numerator and denominator is then assigned to each case in the study sample and averaged across all the cases.
Household profile rate (PROR) = 0.568
The study profile rate is computed as an average of the cohort profile rates for all households in the study sample. Although the average number of profiled panel members per household is usually greater than 1, a household is considered "profiled" when at least one member completes a profile survey. In this study, an overall mean of 56.8% of recruited households successfully completed a profile survey.
Study completion rate (COMR) = 0.845
For this particular study only one panel member per household was selected at random to be part of the study sample. At the end of the fielding periods, 84.5% of assigned cases completed the study survey. (Note: Substitution, i.e., another member of the same household taking the survey instead of the sampled respondent, was not allowed in this study. This is also the general policy for KnowledgePanel samples.)
Break-off rate (BOR) = 0.0056
Among all people who started the survey, 0.56% did break off before the interview was completed. These cases are therefore considered partial interviews.
Household retention rate (RETR) = 0.390
The retention rate is computed as an average of the cohort retention rates for all members in the study sample.
Cumulative response rate 1 (CUMRR1) = (0.326 x 0.568 x 0.845) = 0.1564 x 100% = 15.64%
Because for this study one member per household was selected in computing the cumulative response rate, we use the household recruitment rate multiplied by the household profile rate and the survey completion rate.
Cumulative response rate 2 (CUMRR2) = (0.326 x 0.568 x 0.845 x 0.390) = 0.0610 x 100% = 6.1%
In the cumulative response rate 2, retention is taken into account.
Dennis, Michael J., and Li Rick. 2007. "More Honest Answers to Surveys? A Study of Data Collection Mode Effects." Interactive Marketing Research Organization's (IMRO), Journal of Online Research. Available at http://ijor.mypublicsquare.com/view/more-honest-answers.
Smith, Tom, and Michael J. Dennis. 2008. "Mode Effects on In-person and Internet Surveys: A Comparison of the General Social Survey and Knowledge Networks surveys." Paper presented at the Joint Statistical Meeting, Section on Survey Research Methods, Denver, CO.
|
Footnotes |
|---|
MARIO CALLEGARO AND CHARLES DISOGRA are with Knowledge Networks, Inc. Previous versions of the paper were presented at the annual Pacific chapter of the American Association for Public Opinion Research conference (AAPOR) in San Francisco in December 2007, at the Political Psychology Research Group at Stanford University in March 2008, and at the 63rd AAPOR Annual conference in May 2008. The authors appreciated the suggestions of these audiences. The authors thank Mike Dennis, Erica Demme, Fran Featherston, Erlina Hendarwan, Ana Villar, and Tom Wells for their useful comments. Willem Saris gave us information about the now discontinued Dutch Telepanel and Marcel Das provided with details of the CentERdata CentERpanel and LISS. We are also very grateful to the two anonymous reviewers and the valuable suggestions made by the POQ editors.
1 In 2006, AAPOR added a section on their standards to compute response rates for Web surveys of specifically named individuals. 
3 http://www.rand.org/labor/roybalfd/american_life.html.
4 During the writing of this paper, we became aware of two other probability-based Web panels: the German Omninet Forsa (http://www.forsa.com) and the Canadian Probit (http://www.probit.ca). Probit uses a mixed method data collection where non-Internet households are surveyed by telephone.
|
References |
|---|
American Association for Public Opinion Research. Final Dispositions of Case Codes and Outcomes Rates for Surveys (2006) 4th ed. Lenexa, KS: AAPOR.
Arbeitskreis Deutscher Markt. Standards for Quality Assurance for Online Surveys. (2001).
Arens Zachary, Steiger Darby Miller. Time in Sample: Searching for Conditioning in a Consumer Panel. Public Opinion Pros (2006) August.
Clinton Joshua D. Panel Bias from Attrition and Conditioning: A Case Study of the Knowledge Networks Panel. (2001) Paper presented at 56th Annual Conference of the American Association for Public Opinion Research, May, Montreal, Canada.
Comley Peter. Online Market Research. In: Market Research Handbook—ESOMAR, ed. (2007) Hoboken, NJ: Wiley. 401–20.
Couper Mick. Web Surveys. A Review of Issues and Approaches. Public Opinion Quarterly (2000) 64(4):464–94.[CrossRef][Web of Science][Medline]
Couper Mick. Issues of Representation in Ehealth Research (with a Focus on Web Surveys). American Journal of Preventive Medicine (2007) 32(5S):S83–9.[CrossRef][Web of Science][Medline]
Couper Mick, Kapteyn Arie, Schonlau Matthias, Winter Joachim. Noncoverage and Nonresponse in an Internet Survey. Social Science Research (2007) 36(1):131–48.[CrossRef][Web of Science]
Couper Mick, Dominitz Jeff. Using an RDD Survey to Recruit Online Panel Members. (2007) Paper presented at 2007 Biennial Conference of the European Survey Research Association, June 25–29, Prague.
DiSogra Charles, Slotwiner Daniel, Clinton Sarah, Chan Elisa, Hendarwan Erlina, Zheng Wei. Nonresponse Bias in Two Methods of Panel Recruitment. (2007) Paper presented at Joint Statistical Meetings (JSM), July 29–August 2, Salt Lake City, UT, USA.
European Federation of Associations of Market Research Organizations. EFAMRO—Quality Standards for Access Panel (QSAP). (2004) Available at: http://www.efamro.com/shortprint2.html (accessed September 2008).
European Society for Opinion and Marketing Research. Conducting Market and Opinion Research Using the Internet. (2005) Available at: http://www.esomar.org/uploads/pdf/ESOMAR_Codes&Guideline-Conducting_research_using_Internet.pdf (accessed September 2008).
European Society for Opinion and Marketing Research. 26 Questions to Help Researchers Buyers of Online Samples. (2008) Available at http://194.38.169.84/uploads/pdf/professional-standards/26questions.pdf (accessed September 2008.
Eysenbach Gunther. Improving the Quality of Web Surveys: The Checklist for Reporting Results from Internet E-Surveys (Cherries). Journal of Medical Internet Research (2004) 6(3):e34.[CrossRef][Medline]
Ezzati-Rice Trena M., Frankel Martin R., Hoaglin David C., Loft John D., Coronado Victor G., Wright Robert A. An Alternative Measure of Response Rate in Random-Digit-Dialing That Screen for Eligible Subpopulations. Journal of Economic and Social Measurement (2000) 26(2):99–109.
Fricker Ronald D., Schonlau Matthias. Advantages and Disadvantages of Internet Research Surveys: Evidence from the Literature. Social Science Computer Review (2002) 14(4):347–67.
Göritz Anja S. Using Online Panels in Psychological Research. In: The Oxford Handbook of Internet Psychology—Joinson Adam N., McKenna Katelyn Y. A., Postmes Tom, Reips Ulf-Dietrich, eds. (2007) Norfolk: Oxford University Press. 473–85.
Groves Robert M. Nonresponse Rates and Nonresponse Bias in Household Surveys. Public Opinion Quarterly (2006) 70(5):646–75.
Groves Robert, Peytcheva Emilia. The Impact of Nonresponse Rates on Nonresponse Bias. A Meta-Analysis. Public Opinion Quarterly (2008) 72(2):167–189.
Hoogendoorn Adriaan W., Daalmans Jacco. Nonresponse in the Recruitment of an Internet Panel Based on a Probability Sample. (2008) Discussion Paper 08007. Statistics Netherlands, Voorburg/Heerlen.
Huggins Vicki, Eyerman Joe. Probability Based Internet Surveys: A Synopsis of Early Methods and Survey Research Results. (2001) Paper presented at Federal Committee on Statistical Methodology Research Conference, November 14–16, Arlington, VA, USA.
Interactive Marketing Research Organization. IMRO Guidelines for Best Practices in Online Sample and Panel Management. (2006) Available at http://www.imro.org/pdf/IMRO_Guidelines_for_Best_Practices_in_Online_Sample_and_Panel_Management.pdf (accessed September 2008).
Lee Sunghee. An Evaluation of Nonresponse and Coverage Errors in a Prerecruited Probability Web Panel Survey. Social Science Computer Review (2006) 24(4):460–75.
Lozar Manfreda Katja, Vehovar Vasja. Survey Design Features Influencing Response Rates in Web Surveys. (2002) Paper presented at International Conference on Improving Surveys, August 25–28, Copenhagen, Denmark.
Lozar Manfreda Katja, Vehovar Vasja. Internet Surveys. In: International Handbook of Survey Methodology—De Leeuw Edith, Hox Joop, Dillman Don, eds. (2008) New York: Lawrence Erlbaum. 264–84.
Miller Jeff. Online Marketing Research. In: The Handbook of Marketing Research. Uses, Abuses and Future Advances—Grover Rajiv, Vriens Marco, eds. (2006) 110–31. Thousand Oaks, CA: Sage.
Office of Management and Budget (OMB). Questions and Answers When Designing Surveys for Information Collections (2006) Washington, DC: OMB.
Pineau Vicki J., Nukulkij Poom, Tang Xiuli. Assessing Panel Bias in the Knowledge Networks Panel: Updated Results from the 2005 Research. In: Joint Statistical Meeting 2005 Proceedings [Cd-Rom] (2006) Alexandria, VA. American Statistical Association. 3480–6.
Postoaca Andrei. The Anonymous Elect. Market Research through Online Access Panels (2006a) Berlin: Springer.
Postoaca Andrei. Response Rates. Avoiding the Red Herrings. In: Panel Research 2006 (2006b) Amsterdam: ESOMAR.
Pratesi Monica, Lozar Manfreda Katja, Biffignardi Silvia, Vehovar Vasja. List-Based Web Surveys: Quality, Timeliness, and Nonresponse in the Steps of the Participation Flow. Journal of Official Statistics (2004) 20(3):451–65.
Reips Ulf-Dietrich. The Methodology of Internet-Based Experiments. In: The Oxford Handbook of Internet Psychology—Joinson Adam N., McKenna Katelyn Y. A., Postmes Tom, Reips Ulf-Dietrich, eds. (2007) Norfolk: Oxford University Press. 373–90.
Saris Willem E. Computer Assisted Interviewing (1991) Newbury Park, CA: Sage.
Saris Willem E. Ten Years of Interviewing without Interviewers: The Telepanel. In: Computer Assisted Survey Information Collection—Couper Mick, Baker Reginald P., Bethleem Jelke, Clark Cynthia Z. F., Martin Jean, Nicholls William L. II, O’Reilly James M., eds. (1998) New York: Wiley. 409–29.
Sayles Harlan, Arens Zachary. A Study of Panel Member Attrition in the Gallup Panel. (2007) Paper presented at 62nd Annual Conference of the American Association for Public Opinion Research, May 17–20, Anaheim, CA, USA.
Schlengen William E., Caddell Jesta M., Ebert Lori, Jordan Kathleen B., Rourke Kathrin M., Wilson David, Thalji Lisa, Dennis Michael J., Fairbank John A., Kulka Richard A. Psychological Reactions to Terrorist Attacks. Findings from the National Study of American's Reactions to September 11. Journal of the American Medical Association (2002) 288(5):581–8.
Schonlau Matthias, VanSoest Arthur, Kapteyn Arie, Couper Mick. Selection Bias in Web Surveys and the Use of Propensity Scores. (2006) Available at http://www.rand.org/pubs/working_papers/2006/RAND_WR279.pdf (accessed September 2008).
Schonlau Matthias, Fricker Ronald D., Elliott Marc N. Conducting Research Surveys Via E-Mail and the Web (2002) Santa Monica, CA: RAND.
Sikkel Dirk, Hoogendoorn Adriaan. Panel Surveys. In: International Handbook of Survey Methodology—De Leeuw Edith, Hox Joop, Dillman Don, eds. (2008) New York: Lawrence Erlbaum. 479–99.
Smith Tom W. Developing Nonresponse Standards. In: Survey Nonresponse—Groves Robert, Dillman Don, Eltinge John L., Little Roderick J. A., eds. (2002) New York: Wiley. 27–40.
Toepoel Vera, Das Marcel, van Soest Arthur. Design of Web Questionnaires: The Effects of the Number of Items per Screen." Field Methods. (in press).
Tortora Robert. Attrition in Consumer Panels. In: Methodology of Longitudinal Surveys—Lynn Peter, ed. (2009) Hoboken, NJ: Wiley. 235–49.
Tourangeau Roger. Web-Based Data Collection. In: Survey Automation. Report and Workshop Proceedings—Cork Daniel L., Cohen Michael L., Groves Robert, Kalsbeek William, eds. (2003) Washington, DC: National Academies Press. 183–98.
Vis Corrie. Who Wants to Become an Online Panel Member? Effects of Respondent Characteristics on the Recruitment of an Online Panel. (2007) Paper presented at 2007 Biennial Conference of the European Survey Research Association, June 25–29th, Prague.
Vonk Ted, van Osenbruggen Robert, Willems Pieter. The Effect of Panel Recruitment and Management on Research Results. A Study across 19 Online Panels. In: Panel Research 2006 (2006) Amsterdam: ESOMAR.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  S. Ansolabehere and D. M. Konisky Public Attitudes Toward Construction of New Power Plants Public Opin Q, September 1, 2009; 73(3): 566 - 577. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||