The Role of Topic Interest in Survey Participation Decisions
Robert M. Groves is at the University of Michigan and the Joint Program in Survey Methodology (JPSM). Stanley Presser and Sarah Dipko are at the University of Maryland and the JPSM. The authors thank the interagency advisory committee for the 2000 Joint Program in Survey Methodology practicum (B. K. Atrostic, Susan Schecter, and Clyde Tucker), the JPSM students in the practicum, Steve Heeringa, Roger Tourangeau, Sonja Ziniel, and two anonymous reviewers, all of whom were helpful in preparing the article.
Address correspondence to Robert M. Groves (email: bgroves{at}isr.umich.edu) or Stanley Presser (e-mail: spresser{at}survey.umd.edu).
 |
Abstract |
|---|
 TOP Abstract IntroductionResearch DesignFinal Sample Dispositions and...ResultsDiscussionSummary and ConclusionsAppendixReferences |
|---|
While a low survey response rate may indicate that the risk of nonresponse error is high, we know little about when nonresponse causes such error and when nonresponse is ignorable. Leverage-salience theory of survey participation suggests that when the survey topic is a factor in the decision to participate, noncooperation will cause nonresponse error. We test three hypotheses derived from the theory: (1) those faced with a survey request on a topic of interest to them cooperate at higher rates than do those less interested in the topic; (2) this tendency for the "interested" to cooperate more readily is diminished when monetary incentives are offered; and (3) the impact of interest on cooperation has nonignorability implications for key statistics. The data come from a three-factor experiment examining the impact on cooperation with surveys on (a) five different topics, using (b) samples from five different populations that have known attributes related to the topics, with (c) two different incentive conditions.
|
Introduction |
|---|
|
TOPAbstractIntroductionResearch DesignFinal Sample Dispositions and...ResultsDiscussionSummary and ConclusionsAppendixReferences |
|---|
Response rates have traditionally been used as indicators of survey quality, based on the fact that nonresponse error is partially a function of the response rate. Recently, three studies have challenged this practice (Curtin, Presser, and Singer 2000; Keeter et al. 2000; Merkle and Edelman 2002). Each found little relation between variation in response rates and changes in nonresponse error. Since nonresponse error in survey statistics such as the unadjusted sample mean is a function of the response rate and the nonrespondents’ characteristics, these studies suggest that respondents and nonrespondents often do not differ on commonly estimated statistics. Thus, research is needed to identify the circumstances under which nonresponse produces nonresponse error. Such identification will be facilitated by theories that distinguish between causes of participation that are related to statistics produced by a survey and causes that are unrelated to those statistics.
One effort along these lines is leverage-salience theory (Groves, Singer, and Corning 2000), which posits that people vary in the importance they assign to different aspects of a survey request. For example, for some individuals, the topic may be important; for others, whether a reputable organization is conducting the survey may be significant; and for still others, a chance to receive a cash reward may be of consequence. According to the theory, the influence of each component of the request depends both on the weight accorded it by a sampled individual (leverage) and on its prominence in the request protocol (salience).
The idea that people differ in their reactions to the features of a survey request is consistent with a diverse set of findings: for instance, that monetary incentives have different effects for different groups (Kulka 1994; Singer 2002), that interviewers who tailor their introductions to the concerns of the respondent obtain higher response rates (Groves and Couper 1998; Morton-Williams 1993), that some individuals respond more readily to a survey conducted by a government agency than by a university (National Academy of Sciences 1979), and that people with more involvement with a survey topic respond at higher levels than those with less involvement (Goyder 1987).
Of the many features of a survey request, we believe that subject matter, or topic, is particularly likely to lead to "nonignorable" nonresponse (Rubin 1987), that which produces nonresponse error. This follows from the fact that people more interested in the topic tend to have attributes on the key survey variables different from those of people less interested in the topic. Hence, statistics computed on variables central to the topic are apt to be among those most susceptible to nonresponse error, especially when the topic is made salient in the recruitment protocol.
Leverage-salience theory does not simply predict that persons interested in the survey topic will be overrepresented among respondents (and underrepresented among nonrespondents), relative to those uninterested. It predicts that the degree of overrepresentation will be a function of the salience of (and attitude toward) the survey topic among those deciding whether to cooperate, relative to the salience of (and attitudes toward) the other factors that are part of the survey request. If there are no other positive features to participation, the effect of topic should dominate the decision. If there are other influences toward participation, then the topic effect should be diminished. Indeed, if the information considered by the sample persons does not include the survey topic, we might expect little difference between respondents and nonrespondents on variables related to the topic.
This perspective on the survey participation decision is compatible with a major social psychological theory of persuasion (Petty and Cacioppo 1986). The theory specifies two principal routes to decision making in reaction to a persuasion attempt. If the topic is of interest, people tend to engage in extensive cognitive processing of the message itself. If the topic is not of interest, people tend to rely on peripheral cues (e.g., characteristics of the speaker as opposed to the message) to make a judgment. This suggests, as does leverage-salience theory, that people uninterested in the topic of a survey are more apt to be influenced by other aspects of the survey request (e.g., the appeal of the interviewer or the offer of a monetary incentive) in deciding whether to participate.
Such a pattern has in fact been reported by Baumgartner and Rathbun (1997), who found that a monetary incentive increased cooperation more among those less interested in the survey topic than it did among the more interested, and by Groves, Singer, and Corning (2000), who found that such an incentive affected cooperation with a community survey more among those who were uninvolved in community affairs than among others. Thus, we expect that a survey on any specific topic will recruit more persons uninterested in the topic when it offers a monetary incentive. In other words, the addition of a monetary incentive (a non-topic-related reason for participation) should dampen the link between interest in the topic and cooperation.
The ideal research design to test this hypothesis would pose a series of independent survey requests to persons whose interest levels on the survey topics were known. The survey requests would make salient only one attribute of the survey—its topic; no other features of the request (e.g., sponsor, length of interview, interviewer characteristics) would be apparent. One request would be for a survey on the topic of greatest interest; another request would be for a survey on the topic of second-most interest; and so on. All the survey requests would be repeated in independent trials with and without a monetary incentive. We would hypothesize that the propensity to cooperate would be a positive function of both topic interest and incentive level and that there would be a negative interaction effect between topic interest and incentive.
The challenges of this ideal design are essentially insurmountable. Truly independent repeated survey requests of the same subject are impossible, so a between-subjects design is needed. Direct measures of people’s interest sets are difficult to obtain, so indirect measures, such as membership on a list that suggests an interest, are a logical compromise (in essence using a binary indicator of interest, although it is inherently a continuous variable). Making topic the only feature of a survey that is salient in the introduction is impractical (given common guidelines on informed consent), so other features, such as interview length, sponsor, and the interviewer will inevitably have effects. Each of these departures from the ideal design makes it more difficult to observe the hypothesized relationship between topic interest and response propensity. Thus, the test of the hypothesis we will report in this article is likely to be a conservative one.
We carried out a randomized factorial experimental design that exposed different populations with known characteristics to telephone survey requests on different topics. To test the hypothesized moderating effects of incentives, random half-samples were sent monetary incentives via an advance letter. We chose survey topics that appeared related (i.e., of interest) to the different populations. We hypothesized that when the topic of the survey was related to known characteristics of sample persons, their response rates would be higher than when the topic of the survey was not directly related to those characteristics. Further, we tested whether offering a prepaid incentive would reduce the effect of topic interest. Finally, we measured the impact of these factors on estimates from the survey, thereby assessing the nature of the nonresponse error associated with response rate differences.
|
Research Design |
|---|
|
TOPAbstractIntroductionResearch DesignFinal Sample Dispositions and...ResultsDiscussionSummary and ConclusionsAppendixReferences |
|---|
SAMPLE DESIGN
The sample for the study was drawn from five separate frames: four list samples representing four different groups, and a random digit dial (RDD) sample intended as a control (to contrast response propensities of the list frames to those of the general telephone household population). We hoped to obtain a total of 2,500 completed interviews, with approximately 500 from each frame, and thus we designed the sample to yield about 4,000 eligible households, 800 from each of the frames (see table 1).
|
There were four sample replicates released during the course of data collection. Each of the four replicates consisted of two randomly assigned parts: letter and non-letter. For cases assigned to the letter condition, advance letters with a study description and a five-dollar bill were mailed one week prior to the sample’s release. Letters were mailed to one-half of the households from the four list samples and to one-half of the households from the RDD frame for which mailing addresses could be obtained. The letters mentioned only the survey topic that matched the randomly assigned introduction condition for the household. In retrospect, we realize that mentioning the survey topic in the letter could have indirectly acted to increase the salience of the topic. Thus, our test of the hypothesis that monetary incentives reduce the impact of salience on cooperation is a conservative one. (In all tables and analyses in this paper, RDD cases without mailing addresses are omitted.)
Given the design of the experiment, the selection of the frames suggested the topics of the survey. The four sampling frames were elementary and secondary schoolteachers (for whom we assumed an interest in education and schools), parents of children 0–6 months old (for whom we assumed an interest in child care and problems of parents), contributors to several presidential candidates in the year 2000 (for whom we assumed an interest in voting and participation in elections), and people aged 65 and older (for whom we assumed an interest in Medicare and health). The fifth, RDD, sample was selected from a one-plus list-assisted frame, purchased from Genesys. A fifth survey topic, assumed to be relatively content-free, was used as a control topic (i.e., "important issues facing the nation"). All five sample frames were restricted to the contiguous United States. The five topics—education and schools, child care and problems of parents, voting and participation in elections, Medicare and health, and important issues facing the nation—were randomly assigned across the five samples in equal proportions. That is, subsamples from all five frames were assigned at random to the five different topics.
Although the goal for the four list samples was to speak to the person on the list, we wanted to make the within-household selection process appear random to both interviewers and respondents. This was done to avoid questions that might arise from asking for a named individual and to blind interviewers to the origin of the sample. To accomplish this for the four list samples, the gender of the target respondent was identified in advance; when no such determination was possible, a random assignment of gender was used. Random assignment of gender was also used for the entire RDD sample. In the actual selection procedure, the interviewers asked to interview either a male or female head of the household. In the event that there was no such person, interviewers were allowed to switch and ask for the other gender.
ELEMENTARY AND SECONDARY SCHOOLTEACHERS
A frame of schoolteachers was obtained from a commercial data base list broker, Direct Mail and Computer Systems. The list owner creates this frame every year in the early spring, starting with a list of people who purchase products for classroom materials. Information from other sources is merged into the list to compile a "teachers at home" database. Our sample was drawn from a list with 955,001 entries: 555,403 from elementary schools and 399,598 from high schools. Although it was not possible to specify public versus private school as part of the sample order, the list broker claimed that public school teachers comprised approximately 95 percent of the database. (Later analysis of respondent characteristics from the survey indicated that the frame included some school employees who were not teachers.)
The database owner was asked to sort the list by school type (primary versus secondary) and the first three zip code digits before selecting 10,000 records (the minimum size order). Following the selection, records without telephone numbers were sent to Donnelly Marketing to have that information appended. Then prior to drawing the final sample from the 5,033 teacher records with names, addresses, and telephone numbers, the file was sorted by school type (primary versus secondary), gender (if available), and first three zip code digits.
PARENTS OF CHILDREN 0–6 MONTHS OLD
A list of parents of recently born children was also accessed through Direct Mail and Computer Services. It consisted of 1.1 million households with children 0–6 months old and was compiled from public birth records as well as from requests for diaper service and other products for infants. Some records included only the mother’s name, others included only the father’s name, and some contained both.
We instructed the source to sort the data base by the newborn’s date of birth (month and year) and first three zip code digits. Following the selection of 10,000 records (the minimum size order), those without telephone numbers were sent to Donnelly Marketing to have that information appended. Before selecting the final sample from the 4,861 records that had telephone numbers, the file was sorted by the child’s date of birth (month and year), gender of parent’s name (male name, female name, or both), and first three zip code digits.
PERSONS 65 AND OLDER
The source for the 65 and older frame was Survey Sampling, Inc. (SSI), which maintains a database of listed telephone households with information on age, drawn from sources such as driver’s license, motor vehicle, and voter registration records. The SSI frame consisted of 369,868 households (in the contiguous United States), with one or more persons 65 and older.
We ordered 2,500 records to ensure adequate sample size due to an expected higher nonworking and nonresidential rate than for the other list samples. The sample was drawn using SSI’s standard "probability proportional to size" by state selection procedures. Prior to drawing the subsample for the experiment, the 2,500 records were sorted by gender, date of birth, and first three zip code digits.
CAMPAIGN CONTRIBUTORS TO NON-MAINSTREAM U.S. PRESIDENTIAL CANDIDATES IN THE YEAR 2000
Presidential candidates who wish to obtain federal matching funds are required by the U.S. Federal Election Commission (FEC) to report information on contributors to their campaigns. This information is posted quarterly on the FEC Web site, and it includes each individual contributor’s full name, address, occupation, and employer, as well as the date and dollar amount of the contribution and a cumulative total of contributions by that individual.
We wanted to sample contributors with high interest in politics and strong commitments to their political beliefs. We expected the greatest intensity of interest would be among contributors to candidates with little chance of winning. For this reason we sampled only contributors to non-mainstream presidential candidates. The candidates chosen were Pat Buchanan, Alan Keyes, and Lyndon Larouche, thus ensuring a mix of contributors from different parts of the political spectrum.
The frame consisted of 9,350 individual records with full names and mailing addresses. Nearly half (43.5 percent) were contributors to Buchanan, roughly one-third (31.0 percent) were Keyes contributors, and the remainder (25.4 percent) were Larouche contributors. The names and addresses of all records were submitted to Telematch to obtain telephone numbers. Over half of the records (60.5 percent) yielded a telephone number, leading to a final sampling frame of 5,656 individuals. The telephone match rate did not differ substantially by candidate, so the final sampling frame was 42 percent Buchanan contributors, 32.5 percent Keyes contributors, and 25.5 percent Larouche contributors. Prior to drawing the sample, the frame was sorted by presidential candidate, quarters in which contributions were made, and the contributor’s first three zip code digits.
RANDOM DIGIT DIAL SAMPLE
A one-plus list-assisted RDD sample of 2,000 telephone numbers was obtained from Genesys, and all records were submitted to Telematch to obtain addresses.
QUESTIONNAIRE STRUCTURE
The samples from each frame were divided into five equal-sized subsamples. Each subsample was assigned to a different survey introduction. The voting introduction, for example, was:
Hello, my name is ——— from the University of Maryland Survey Research Center. Here at the university we are doing a study on important issues such as voting and participation in elections.
We are gathering information on people’s experiences and opinions about voting and participation in elections.
For your household we have randomly selected the [male/female] head of the household. May I please speak with [him/her]?
The four other introductions replaced "voting and participation in elections" with "education and schools," "child care and problems of parents," "Medicare and health," or "important issues facing the nation."
A key design decision was how salient to make the topic of the survey. While the force of the experimental manipulation of topic could have easily been increased by having the interviewer give more elaborate descriptions, we chose to create the kinds of brief introductions that are commonly used in household telephone surveys.
The order in which the questions about the four topics were asked was varied by condition. For the four specific topic survey introductions, the questionnaire’s opening section matched the introduction topic. When the "issues facing the nation" version of the introduction was used, the questionnaire was randomly assigned to start with one of the four topical sections.
INTERVIEWERS
The interviewers included twelve first-year students from the Joint Program in Survey Methodology (JPSM) master’s program as well as the regular interviewing staff of the University of Maryland Survey Research Center (SRC). Roughly 10 percent of all call attempts and 15 percent of completed interviews were conducted by the JPSM students during the first three weeks of March 2000. The regular SRC interviewers were primarily undergraduate students, and their work was conducted over the course of three months, from mid-February through early May 2000.
|
Final Sample Dispositions and Response Rates |
|---|
|
TOPAbstractIntroductionResearch DesignFinal Sample Dispositions and...ResultsDiscussionSummary and ConclusionsAppendixReferences |
|---|
Of the 4,070 total telephone numbers from all five sample frames, 369 were non-households, including businesses and nonworking numbers. Non-household numbers ranged from a low of 6.4 percent among the contributor sample to 12.1 percent among parents of newborns, compared to 17.1 percent among RDD cases.
Of the 3,701 remaining telephone numbers, 835 were refusals and break-offs, 181 posed difficulties such as a language other than English, illness, or hearing problems, 311 were noncontacts in which a respondent was never selected (e.g., the telephone was always answered by a household answering machine) or the selected respondent was never reached for an interview, and 75 were never answered after at least 20 callbacks. A total of 2,330 interviews were completed (of which 31 were partial interviews), leading to an overall response rate of between 63.0 and 63.4 percent, depending on the proportion of never-answered numbers that are treated as eligible. The separate results for each sample frame are shown in table 2.
|
Compared to the special population list frames, the RDD frame suffers from a higher noncontact rate (due to the indeterminacy of never-answered numbers), and thus the cooperation rate (AAPOR COOP2) is a useful comparative measure. The RDD cooperation rate of 59.7 percent is similar to that of the 65 and older frame (61.6 percent), both of which are much lower than those of the other frames. The political contributors exhibit the highest cooperation rates (79.5 percent).
|
Results |
|---|
|
TOPAbstractIntroductionResearch DesignFinal Sample Dispositions and...ResultsDiscussionSummary and ConclusionsAppendixReferences |
|---|
THE DEPENDENT VARIABLE
Cases in which the introduction was never heard are not likely to show an effect of our main experimental manipulation. Consequently, our main analyses include only those cases where a contact was made with someone in the household, and the interviewer was able to deliver the introduction at least as far as mentioning the survey topic. The dependent variable is the outcome of the first contact with the sampled household during which the survey topic was mentioned. (We chose that outcome as one that is minimally affected by differential skills of interviewers and other potential influences not key to the research design.) The variable is coded "1" if the interview was begun on that call, and "0" if it was not. Thus, completed and partial interviews on the first contact are contrasted with other outcomes of that contact such as refusals, appointments, and unavailable respondents. (Cases in which all contacts with the household yielded a language barrier or other respondent problem, such as deafness, are excluded from our analyses.) The fact that the first contact in which the introduction was read may have occurred with a household member other than the targeted respondent is another reason our tests of the salience hypothesis are conservative.
CONTRASTING FRAMES, TOPICS, AND INCENTIVE TREATMENTS
A two-factor display of the dependent measure, the cooperation rate at first exposure to the stimulus, is presented in table 3 for the five frame populations and the five survey topics. The table contains both the final response rates for each experimental treatment cell and the percentage of sample persons who cooperated on the same contact on which the survey introduction was first given. The political contributors had the highest average cooperation rate among the different frame populations (54.8 percent) and the RDD frame, the lowest (34.1 percent). The various survey topics exhibit smaller differences in level of cooperation. "Education and schools" has the highest rate (47.5 percent); "voting and elections" and the control topic "issues facing the nation" have the lowest (about 41 percent). These main effects of frame and topic are not the focus of our research, but they suggest how response rates differ across common topics and populations.
|
Table 3 permits an initial look at our first hypothesis. The table row "Average of irrelevant topics" shows the average cooperation among the four survey topics judged not relevant to the given frame. For example, for the sample of teachers, the average cooperation rate for the four other survey topics was 42.0 percent, compared to 55.9 percent for the "education and schools" topic (p < .05). The 13.9 percentage point difference supports the hypothesis that persons will cooperate with surveys on topics that interest them more than on other topics. Similar support for the topic interest hypothesis appears for new parents and people 65 and older. The political contributors, possibly because all of the topics may be seen as having political content (admittedly a post hoc hypothesis), show no preference for the "voting and elections" topic over the others.
Leverage-salience theory predicts that the treatment effects should be smaller for the response rate (which reflects the impact of callbacks and refusal conversion designed in part to offset respondents’ lack of interest in the topic) than for the cooperation rate at the first contact. For the teacher frame, as expected, the differences in response rates due to topic relevance are smaller than the differences in cooperation rates (9.9 percentage points versus. 13.9 percentage points; t = 9.55, df = 748, p < .001). For other frames the evidence is more mixed. This could reflect a lack of diminished salience of the topic stimulus, relative to the effects of callbacks and interviewer persuasion, which involve survey features in addition to the topic (e.g., sponsorship, importance of respondent cooperation, interviewer professionalism).
THE EFFECTS OF MONETARY INCENTIVES
We expected the topic interest effects we observed for three of the four frames to be diminished by the provision of the $5 incentive. Table 4 shows that this occurs in two of the three cases: teachers and individuals 65 and older. Surprisingly, however, the reverse occurs for new parents.
|
MULTIVARIATE MODELS
We then constructed multivariate models that simultaneously included the effects of topic and frame membership in an equation predicting the likelihood of cooperation at the first contact. The specification of the model is
 |
We fit the model in two steps. First, we examine whether, in the presence of the multivariate controls, the effect of the topic interest indicators remains. These results are listed in the column of table 5 labeled "topic interest model." The test statistic for the multiple contrast hypothesis involving all the ßm > ßn contrasts is significant at the .001 level (
2 = 10.96, df = 1). The estimated increase in the odds of cooperating with a relevant topic (pooling all cases of relevant topic together) versus all others is 38 percent (a marginal odds ratio of 1.378). Thus, the effect of topic interest on this indicator of cooperation is substantial.
|
Step two of the model-fitting process addresses the second hypothesis: that when incentives are offered, the higher cooperation rates of those interested in the topic are diminished, reflecting the stronger effects of incentives on those not interested in the topic of the survey. This hypothesis is tested by fitting interaction terms that reflect the combined effects of incentive and topic interest.
 |
2 = 12.07, df = 12, p = .44; data not shown).
An examination of the different sampling frames bears out the bivariate result from table 4 that the new parents’ frame is an exception with respect to the interaction. When coefficients associated with that frame are omitted, the interaction effect is more pronounced but still nonsignificant (2 = 13.2, df = 9, p = 0.15). We are not certain why the new parents exhibit effects that appear distinctive from those of the other frames. An after-the-fact inquiry to the vendor from whom we purchased our sample revealed that the new parents’ list was a "best-seller," very popular for direct telemarketing to new parents. Thus, it is possible that telephone contacts with strangers about their new birth are disproportionately sales efforts; hence, a stranger requesting an interview relevant to the new birth is precisely what the new parents are not interested in having. In any case, this anomaly needs further investigation.
CONSEQUENCE FOR SURVEY ESTIMATES
To this point we have demonstrated that people who possess characteristics that predispose them to be interested in a particular topic are generally more likely to cooperate with a survey request when that topic is mentioned in the introduction. This has potential implications for nonresponse error of survey estimates. To the extent that nonresponse is caused by interest in the topic, survey results on matters related to interest in that topic should be biased by the overrepresentation of those with high interest (and corresponding underrepresentation of those with low interest). Thus, we hypothesized that the distributions of answers to some questions asked in our survey would vary by introduction. The sample exposed to the voting introduction, for instance, ought to have expressed higher levels of political interest and involvement to the survey questions than the samples that heard the other introductions.
We tested this hypothesis with three kinds of items. First, toward the end of the interview we asked respondents which of our four topics they considered the most important problem facing the nation: improving education, health care for the elderly, child care for preschoolers, or increasing participation in elections. We hypothesized that choice of a topic should be greatest when it was mentioned in the survey introduction, on the assumption that ratings of a topic’s importance are linked to interest in the topic.
Second, from each of the four topic sections of the questionnaire we chose one item that appeared to us most related to concern about the topic: attitude toward teacher salaries, opinion about the difficulty of finding child care, enrollment in Medicare, and attention to the 2000 election campaign. We expected that respondents would say more often that teacher salaries were too high or too low (as opposed to saying "about right" or having no opinion) in the survey about "education and schools"; express more beliefs that high-quality child care was difficult to find in the survey about "child care and problems of parents"; report highest Medicare enrollment in the survey on "Medicare and health"; and claim greatest attention to the campaign in the survey on "voting and election participation." Our assumption was that each of these characteristics is apt to be related to interest in the topic.
Finally, the interview contained questions that allowed us to identify whether respondents were members of the four populations from which we drew our list samples: Are you a teacher? Do you have a child 0 to 6 months old? When were you born? Did you contribute money to a political candidate? Our hypothesis was that membership in a population would be greatest with the corresponding topic introduction, again on the assumption that interest in the topic is promoted by membership on the frame.
Because we drew on answers from the interview, these analyses switched from the total sample (interviewed and never interviewed cases) that we used in preceding analyses to only those cases that were interviewed (either on the first contact or at a later point). Each test used the interviewed sample from four of the five frames: the sample from the frame relevant to a topic was excluded from analyses comparing that introduction to the others. For instance, in comparing the level of contributors (or political interest, or the view that political participation was the most important problem facing the nation) between the politics introduction and the other introductions, the sample from the contributors’ frame was excluded. This was done because that frame’s likely homogeneity on these variables meant that contributors would not be apt to show variation on such variables by introduction.
Table 6 shows the results of these tests. Eight of the twelve tests are in line with the hypothesis. For example, among the four noncontributor frames, more respondents claim to have made a campaign contribution in the "voting and elections" condition than in the other conditions combined. On the other hand, only three of these eight effects are significant at the .05 level; and of the four results not in line with expectation, one is statistically significant. Thus, overall, there is support for the hypothesis, but it is weak.
|
These analyses, however, use data from all respondents, irrespective of whether they initially cooperated or required refusal conversion and irrespective of whether they received a monetary incentive. Yet there is less reason to expect the hypothesis to apply to respondents who were reluctant to participate after hearing the introduction or to those who received the incentive (who had a non-topic-rationale for participation). Thus, we repeated the analyses separately for groups differing in the ease with which the interview was obtained and by incentive condition. We expected that our hypothesis would receive stronger confirmation among those who were easy to interview and among those who did not receive the monetary incentive.
As can be seen in tables 7 and 8, there is not much support for the expectation that either interview ease or the monetary incentive affects the association between introduction and survey result. There is a little evidence of an effect of the introduction, but it is about the same for hard cases and easy ones, incentive cases and non-incentive ones.
|
|
The relative weakness of the results concerning nonresponse error may be partly due to departures from the ideal experimental design that we discussed in the introduction. If the motivation for cooperation is multifaceted, the hypothesized effects are likely to be attenuated. In the case of our design, heterogeneity in topic interest within each frame, the possible influences of sponsorship ("the University of Maryland Survey Research Center"), and the moderating effects of interviewer behavior may explain some of these results.
But there are also other reasons that the impact on survey estimates was not more pronounced. Two of our groups (teachers and recent parents) constitute only tiny fractions of the total population, and the other two (contributors and those 65 and older) are relatively small (on the order of 15 to 20 percent). Moreover, membership in these four populations is not consistently related to our survey variables. Of the eight variables in the lower two panels of tables 6
–8 ("most important problem" choices and "other survey reports"), only four are significantly related to membership in the relevant population, and two of the four involve teachers, who comprise a tiny fraction of the total population. Thus, the effect of topic interest on cooperation does not translate into much of an effect on the survey’s estimates.
AN ANALYTIC DERIVATION
To put the size of these effects into perspective, we examined the maximum possible impact on nonresponse error in the kind of experiment we mounted. The appendix derives analytically the bias of the difference between a statistic from a survey about topic Y, which is of interest to a subset of the population, versus that from a survey about an unrelated topic. It asks the question, for estimates of the percentage of persons in the population with an interest in Y, what would be the bias difference between the result from the relevant topic survey and the irrelevant topic survey? The answer is that the bias is a function of the true percentage of persons with the interest and the difference in response rates to the two surveys for the interested population relative to the response rate difference for the uninterested. On our surveys, the corresponding response rate differences are approximately 10, 7, 14, and 0 percentage points (last row of table 3). With response rate differences generated by topic interest differences in this range, the impact on nonresponse error must be rather small. For the case where the true population percentage is 50 percent, for example, the 7–10 percentage point response rate differences due to topic interest produce biases of less than 5 percentage points around the 50 percent true value. If the population statistic is smaller, say 20 percent, then the bias induced by topic interest is in the range of 1–2 percentage points around the 20 percent. Thus, given our response rate differences, the impact on survey estimates of interest have to be small and will be even smaller for estimates of variables that are not perfectly correlated with interest.
|
Discussion |
|---|
|
TOPAbstractIntroductionResearch DesignFinal Sample Dispositions and...ResultsDiscussionSummary and ConclusionsAppendixReferences |
|---|
The results of the experiment were not uniform over frame and topic. There are at least three alternative explanations for this:
- Leverage-salience theory is incorrect;
- Our frames inadequately operationalized interest; and
- Our survey introductions inadequately made topic a salient decision criterion.
The finding that political contributors cooperate with all the topics (relative to the RDD control group and other frames) raises the possibility that all our topics are of interest to them. This suggests that the operationalization of the topic set yielded a poor test of the hypotheses for political contributors. If we had used a topic of no political relevance, say, sports, the results might have been different, and this would be a useful step for further research. Alternatively, the high cooperation rates for all topics may imply that political contributors have high response rates on all survey topics. In short, they are consistent responders to surveys. The possibility that some subgroup places high positive leverage on the very act of participating in surveys could be incorporated into the theory. If large parts of the population were indifferent to all features of the survey request, however, it would contradict the basic foundation of the theory. Yet this is incompatible with the findings cited at the beginning of the article (that different persons respond under different conditions).
We believe that explanations (b) and (c) are more likely: the frames incompletely identify interest groups, and the survey introduction made the topic insufficiently salient. There are many influences on whether one responds, in addition to topic. Interviewer voices have effects (Oksenberg, Coleman, and Cannell 1986), sponsorship of the survey may act independently of topic (Groves and Couper 1998), and other situational factors act to attenuate the influence of topic on cooperation. This implies that larger effects might be obtained if the introduction had been altered to heighten the salience of topic relative to other factors.
The upshot for practicing survey researchers, however, is good news. We have provided a theoretical and analytic (in the appendix) rationale, with further empirical support, for the conclusions of Curtin, Presser, and Singer (2000), Keeter et al. (2000), and Merkle and Edelman (2002). Leverage-salience theory suggests that there are many diverse influences on survey participation. Only those influences linked to the survey statistics of interest need cause concern to the analyst. Survey designs that induce multiple influences (mode differences, incentives, alternative interviewers) tend to activate alternative causes of participation, thereby increasing the diversity of the respondent pool and reducing the likely nonresponse error. On the other hand, a request that makes survey topic most salient should produce a linkage between nonresponse rates and nonresponse error (e.g., self-administered questionnaires prominently displaying questions on a single topic prior to the sample person’s decision about participation).
|
Summary and Conclusions |
|---|
|
TOPAbstractIntroductionResearch DesignFinal Sample Dispositions and...ResultsDiscussionSummary and ConclusionsAppendixReferences |
|---|
We have tested whether topic interest effects might be one common source of nonignorable nonresponse in household surveys. That is, do people cooperate at higher rates to surveys on topics that interest them, and, if so, to what extent does that behavior affect survey estimates? We crafted a test that attempted to identify sets of people who shared an interest in a specific topic and then asked them to participate in a survey on that topic. The first challenge of such a design is choosing a survey topic such that all members of the specific population do indeed share that interest. A second challenge is crafting a survey introduction that is realistic (i.e., has high external validity) and makes salient the topic of the survey. All of our conclusions are conditioned on these features of our experiment.
Judging from behavior on the first contact containing the survey introduction, we found that persons cooperated at higher rates to surveys on topics of likely interest to them. The odds of cooperating are roughly 40 percent higher for topics of likely interest than for other topics, based on the four frames utilized in the experiment. Given these results and the deductions from leverage-salience theory, we suspect we could make the 40 percent much higher by making the topic a much more salient aspect of the survey introduction. It is important to note that the overall effects on total response rates of these effects are dampened by noncontact nonresponse, as well as by physical-, mental-, and language-caused nonresponse.
The second hypothesis deduced from leverage-salience theory is that by making a monetary incentive a salient feature of the request, those persons attaching high leverage to personal benefits would cooperate at higher rates (in contrast to their behavior without the financial incentive). This implies that surveys with monetary incentives should show lower tendencies for the "interested" to respond at higher rates than others. The hypothesized direction of effect was found, but it did not attain statistical significance.
The final step in our tests of leverage-salience theory addressed the question, "What is the practical import on survey estimates of the effects of topic interest?" The analytic findings in the appendix show that such effects will be a function of (a) the relative size of the subpopulation interested in a given topic (i.e., prevalence of high leverage), and (b) the degree to which the survey request highlights the topic (i.e., saliency of the topic), producing relatively higher response rates for those interested. In our case, these magnitudes were not large enough to generate much bias due to the main effect of topic interest on cooperation.
|
Appendix |
|---|
|
TOPAbstractIntroductionResearch DesignFinal Sample Dispositions and...ResultsDiscussionSummary and ConclusionsAppendixReferences |
|---|
In this appendix we calculate the maximum nonresponse bias that can arise under various conditions given the experiment described in the article. Nonresponse bias only occurs for variables related to nonresponse propensities. Under our theoretical perspective the largest nonresponse bias is apt to occur for those variables that directly cause nonresponse, such as interest in the survey topic. Here we focus on how the estimate of the proportion of persons interested in topic Y differs between two surveys: Survey 1, identified as a study about topic Y and Survey 2, identified as a study about another topic, unrelated to Y.
Under leverage-salience theory, those interested in the topic (Yi = 1, for the i-th person) will be more likely to cooperate with Survey 1 relative both to their own cooperation with Survey 2 and to the cooperation with Survey 1 of those not interested in the topic (Yi = 0). We consider bias in estimating 
in the target population of size N. The estimate of 
from the first survey is the mean among the r1 respondents of Survey 1, or 
. Similarly, the estimate from the second survey is the mean among the r2 respondents of Survey 2, or 
. Of the N persons in the population, 
are persons with an interest in the topic.
A traditional expression for the nonresponse bias of a respondent mean is
 |
is the mean probability of responding, and 
and 
are the means of the survey variable for the respondents and nonrespondents, respectively. For our purposes it is desirable to express the bias as a function of the covariance between the response probability of a case and its value on Y. By using the presentation of Lessler and Kalsbeek (1987, section 3.2), the bias of the respondent mean can be expressed as
 |
pY is the covariance between the response probability and Y, and Y is the population mean of the survey variable. We compute the covariance on the entire population of size N, for which the i-th person manifests p1i likelihood of responding to Survey 1 and p2i likelihood of responding to Survey 2. We first express the difference of nonresponse biases of the two respondent means as
 |
 |
) persons for whom Yi = 0, and the product (p1i – p2i)Yi always equals zero, and NY persons with Yi = 1, for whom the product is always equal to the difference in the response rates between Survey 1 and Survey 2. This is equivalent to
 |
The B term is equivalent to
 |
, ignoring the technical estimation bias of the ratio mean, and similarly for E(
). After applying algebraic transformation,
 |
- the prevalence in the population of interest in the topic, and
- the ratio of the response rate difference between Survey 1 and Survey 2 among the interested group to the response rate difference among the not interested.
We expect that ratio to be higher in Survey 1 than Survey 2, because Survey 1 increases topic leverage for those interested in Y. Hence, we expect that 
and 
. Further, we note that 
and are functions of Y, so that the expression more generally is
 |
To get a sense of the magnitude of possible nonresponse bias differences, we let Y, the proportion interested in Y in the target population, vary from 0.0 to 1.0; we set 
, the response rate for Survey 1 among those interested in Y, at two different levels (90 percent in figure A1 and 60 percent in figure A2); we set the response rate for those uninterested in Y on Survey 1 equal to (a) their response rate on Survey 2 (
), and to (b) the response rate in Survey 2 for those interested in Y (
).
|
|
Figures A1 and A2 plot, on the vertical axis, the magnitude of the bias difference for the two estimates of the proportion of the target population interested in Y (i.e., 
) as a function of the difference in response rates, assuming the average response propensities are all the same for people who are not interested in the topic of the survey (i.e., (
). The response rate difference between Survey 1 and Survey 2 is shown on the left-right axis; the proportion of the target population that is interested in Y is shown on the near-far axis.
When the proportion in the population interested in Y is either 0 or 100 percent, the difference in the bias between the surveys is zero. With increasing distance from 0 and 100 percent, the difference in bias increases, with the maximum bias difference when 50 percent of the population is interested in Y. Thus, the surfaces have curvilinear features.
The graphs show that the maximum nonresponse bias differences due to treatments like those described in this article (where the response rate differences between Survey 1 and 2 for those interested in the topic tended to be less than 15 percentage points) will be in the 2–7 percentage point range. When those interested in Y respond at a very high rate in Survey 1 (e.g., 90 percent as in figure A1), nonresponse bias differences require larger nonresponse rate differences relative to cases where they respond at lower rates in Survey 1 (compare the slope of figure A1 to that of figure A2 on the left-right axis).
In the survey methodological literature there are many experiments that find response rate gains of 5–10 percentage points from some design feature (e.g., incentives, advance letters). Figures A1 and A2 imply that experiments with such outcomes will have relatively little impact on nonresponse error of estimated proportions. Following the analytic results above, this is especially true when (a) those sensitive to the manipulation are either a small minority of the population or the vast majority of the population, and (b) the survey variables of interest are less than perfectly correlated with sensitivity to the design feature found to increase participation.
|
References |
|---|
|
TOPAbstractIntroductionResearch DesignFinal Sample Dispositions and...ResultsDiscussionSummary and ConclusionsAppendixReferences |
|---|
American Association for Public Opinion Research. 2000. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. Ann Arbor, MI: AAPOR.
Baumgartner, Robert, and Pamela Rathbun. 1997. "Prepaid Monetary Incentives and Mail Survey Response Rates." Paper presented at the annual conference of the American Association for Public Opinion Research, Norfolk, VA.
Curtin, Richard, Stanley Presser, and Eleanor Singer. 2000. "The Effects of Response Rate Changes on the Index of Consumer Sentiment." Public Opinion Quarterly 64:413–28.[Abstract]
Goyder, John. 1987. The Silent Minority: Nonrespondents on Sample Surveys. Boulder, CO: Westview Press.
Groves, Robert, and Mick Couper. 1998. Nonresponse in Household Interview Surveys. New York: John Wiley and Sons.
Groves, Robert, Eleanor Singer, and Amy Corning. 2000. "Leverage-Salience Theory of Survey Participation: Description and an Illustration." Public Opinion Quarterly 64:299–308.[CrossRef][Web of Science][Medline]
Keeter, Scott, Carolyn Miller, Andrew Kohut, Robert Groves, and Stanley Presser. 2000. "Consequences of Reducing Nonresponse in a Large National Telephone Survey." Public Opinion Quarterly 64:125–48.[Abstract]
Kulka, Richard A. 1994. "The Use of Incentives to Survey ‘Hard-to-Reach’ Respondents: A Brief Review of Empirical Research and Current Practice." Paper prepared for Seminar on New Directions in Statistical Methodology, Bethesda, MD.
Lessler, Judith, and William Kalsbeek. 1987. Nonsampling Errors in Surveys. New York: Wiley.
Merkle, Daniel, and Murray Edelman. 2002. "Nonresponse in Exit Polls: A Comprehensive Analysis." In Survey Nonresponse, ed. R. M. Groves, D. A. Dillman, J. L. Eltinge, and R. J. A. Little, pp. 243–58. New York: Wiley.
Morton-Williams, Jean. 1993. Interviewer Approaches. Brookfield, VT: Dartmouth.
National Academy of Sciences. 1979. Panel on Privacy and Confidentiality as Factors in Survey Response. Washington, DC: National Academy Press.
Oksenberg, Lois, Lerita Coleman, and Charles F. Cannell. 1986. "Interviewers’ Voices and Refusal Rates in Telephone Surveys." Public Opinion Quarterly 50:97–111.
Petty, Richard E., and John T. Cacioppo. 1986. Communication and Persuasion: Central and Peripheral Routes to Attitude Change. New York: Springer-Verlag.
Rubin, Donald R. 1987. Multiple Imputation for Nonresponse in Surveys. New York: Wiley.
Singer, Eleanor. 2002. "The Use of Incentives to Reduce Nonresponse in Household Surveys." In Survey Nonresponse, ed. R. M. Groves, D. A. Dillman, J. L. Eltinge, and R. J. A. Little, pp. 163–78. New York: Wiley.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  C. Greenlaw and S. Brown-Welty A Comparison of Web-Based and Paper-Based Survey Methods: Testing Assumptions of Survey Mode and Response Cost Eval Rev, October 1, 2009; 33(5): 464 - 480. [Abstract] [PDF]
|
|
|
|
 |
|
 A. Peytchev, R. K. Baxter, and L. R. Carley-Baxter Not All Survey Effort is Equal: Reduction of Nonresponse Bias and Nonresponse Error Public Opin Q, September 2, 2009; (2009) nfp037v1. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. R. Petrolia and S. Bhattacharjee Revisiting Incentive Effects: Evidence from a Random-Sample Mail Survey on Consumer Preferences for Fuel Ethanol Public Opin Q, September 1, 2009; 73(3): 537 - 550. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 W. L. Parsons, P. S. Duke, P. Snow, and A. Edwards Physicians as parents: Parenting experiences of physicians in Newfoundland and Labrador Can Fam Physician, August 1, 2009; 55(8): 808 - 809.e4. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. W. Courser, S. R. Shamblen, P. J. Lavrakas, D. Collins, and P. Ditterline The Impact of Active Consent Procedures on Nonresponse and Nonresponse Error in Youth Survey Data: Evidence From a New Experiment Eval Rev, August 1, 2009; 33(4): 370 - 395. [Abstract] [PDF]
|
|
|
|
 |
|
 J. L. Holland and L. M. Christian The Influence of Topic Interest and Interactive Probing on Responses to Open-Ended Questions in Web Surveys Social Science Computer Review, May 1, 2009; 27(2): 196 - 212. [Abstract] [PDF]
|
|
|
|
|
|
R. K. Goidel, C. M. Freeman, S. Procopio, and C. F. Zewe Who Participates in the "Public Square" and Does It Matter? Public Opin Q, December 1, 2008; 72(4): 792 - 803. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. Kano, T. Franke, A. A. Afifi, and L. B. Bourque Adequacy of Reporting Results of School Surveys and Nonresponse Effects: A Review of the Literature and a Case Study Educational Researcher, November 1, 2008; 37(8): 480 - 490. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. A. Klofstad, S. Boulianne, and D. Basson Matching the Message to the Medium: Results from an Experiment on Internet Survey Email Contacts Social Science Computer Review, November 1, 2008; 26(4): 498 - 509. [Abstract] [PDF]
|
|
|
|
|
|
S. L. Mann, D. J. Lynn, and A. V. Peterson Jr The "Downstream" Effect of Token Prepaid Cash Incentives to Parents on Their Young Adult Children's Survey Participation Public Opin Q, September 1, 2008; 72(3): 487 - 501. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. M. Mellor, R. B. Rapoport, and D. Maliniak The Impact of Child Obesity on Active Parental Consent in School-Based Survey Research on Healthy Eating and Physical Activity Eval Rev, June 1, 2008; 32(3): 298 - 312. [Abstract] [PDF]
|
|
|
|
|
|
R. M. Groves and E. Peytcheva The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis Public Opin Q, June 1, 2008; 72(2): 167 - 189. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. Bosnjak, W. Neubarth, M. P. Couper, W. Bandilla, and L. Kaczmirek Prenotification in Web-Based Access Panel Surveys: The Influence of Mobile Text Messaging Versus E-Mail on Response Rates and Sample Composition Social Science Computer Review, May 1, 2008; 26(2): 213 - 223. [Abstract] [PDF]
|
|
|
|
 |
|
 R. Bautista, M. Callegaro, J. A. Vera, and F. Abundis Studying Nonresponse in Mexican Exit Polls Int. J. Public Opin. Res., December 1, 2007; 19(4): 492 - 503. [Full Text] [PDF]
|
|
|
|
|
|
N. J. Stroud and K. Kenski From Agenda Setting to Refusal Setting: Survey Nonresponse as a Function of Media Coverage Across the 2004 Election Cycle Public Opin Q, December 1, 2007; 71(4): 539 - 559. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. G. Rogelberg and J. M. Stanton Introduction: Understanding and Dealing With Organizational Survey Nonresponse Organizational Research Methods, April 1, 2007; 10(2): 195 - 209. [Abstract] [PDF]
|
|
|
|
 |
|
 H. Roose, J. Lievens, and H. Waege The Joint Effect of Topic Interest and Follow-Up Procedures on the Response in a Mail Questionnaire: An Empirical Test of the Leverage-Saliency Theory in Audience Research Sociological Methods Research, February 1, 2007; 35(3): 410 - 428. [Abstract] [PDF]
|
|
|
|
 |
|
 S. M. Horwitz, K. J. Kelleher, R. E.K. Stein, A. Storfer-Isser, E. A. Youngstrom, E. R. Park, A. M. Heneghan, P. S. Jensen, K. G. O'Connor, and K. E. Hoagwood Barriers to the Identification and Management of Psychosocial Issues in Children and Maternal Depression Pediatrics, January 1, 2007; 119(1): e208 - e218. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. L. Holbrook, I. C. Farrar, and S. J. Popkin Surveying a Chicago Public Housing Development: Methodological Challenges and Lessons Learned Eval Rev, December 1, 2006; 30(6): 779 - 802. [Abstract] [PDF]
|
|
|
|
|
|
H. Schuman and A. D. Corning Comparing Iraq to Vietnam: Recognition, Recall, and the Nature of Cohort Effects Public Opin Q, March 1, 2006; 70(1): 78 - 87. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
E. Singer Introduction: Nonresponse Bias in Household Surveys Public Opin Q, January 1, 2006; 70(5): 637 - 645. [Full Text] [PDF]
|
|
|
|
|
|
R. M. Groves Nonresponse Rates and Nonresponse Bias in Household Surveys Public Opin Q, January 1, 2006; 70(5): 646 - 675. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
T. P. Johnson, Y. I. Cho, R. T. Campbell, and A. L. Holbrook Using Community-Level Correlates to Evaluate Nonresponse Effects in a Telephone Survey Public Opin Q, January 1, 2006; 70(5): 704 - 719. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. M. Groves, M. P. Couper, S. Presser, E. Singer, R. Tourangeau, G. P. Acosta, and L. Nelson Experiments in Producing Nonresponse Bias Public Opin Q, January 1, 2006; 70(5): 720 - 736. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. Olson Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias Public Opin Q, January 1, 2006; 70(5): 737 - 758. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. Keeter, C. Kennedy, M. Dimock, J. Best, and P. Craighill Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey Public Opin Q, January 1, 2006; 70(5): 759 - 779. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. M. Brick, S. Dipko, S. Presser, C. Tucker, and Y. Yuan Nonresponse Bias in a Dual Frame Sample of Cell and Landline Numbers Public Opin Q, January 1, 2006; 70(5): 780 - 793. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. B. Mann Do Advance Letters Improve Preelection Forecast Accuracy? Public Opin Q, December 1, 2005; 69(4): 561 - 571. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. Curtin, S. Presser, and E. Singer Changes in Telephone Survey Nonresponse over the Past Quarter Century Public Opin Q, March 1, 2005; 69(1): 87 - 98. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||