Public Opinion Quarterly - recent issues

Subscriptions

2009-06-12

Editorial Board

2009-06-12

Predictors of Public Support for Gender-Related Affirmative Action: Interests, Gender Attitudes, and Stratification Beliefs

Kane, E. W., Whipkey, K. J. — 2009-06-12

Drawing on data from the General Social Survey during the period from 1996 through 2006, we explore predictors of support for gender-related affirmative action in the United States. Following the literature on race-based affirmative action, we identify three main domains of predictors, each of which also resonates well with themes within scholarship on gender: interests; gender-related attitudes; and general stratification beliefs. In multivariate analyses, at least some predictors within each domain are significant. We conclude that like support for race-based affirmative action, support for gender-based affirmative action is based on a combination of interests, gender attitudes, and general stratification beliefs, but that gender attitudes are less important in shaping such support than racial beliefs are in shaping support for race-based affirmative action. The implications of these findings for the literature on race-related affirmative action as well as for the literature on gender-related attitudes are considered.

Impact of T-ACASI on Survey Measurements of Subjective Phenomena

2009-06-12

Numerous studies have shown that audio-computer-assisted self-interviewing (audio-CASI) and telephone audio-CASI (T-ACASI) technologies yield increased reporting of sensitive and stigmatized objective phenomena such as sexual and drug use behaviors. Little attention has been given, however, to the impact of these technologies on the measurement of subjective phenomena (attitudes, opinions, feelings, etc.). This article reports results for the seven subjective measurements included in the National STD and Behavior Measurement Experiment (NSBME). NSBME drew probability samples of USA and Baltimore adults (Ns = 1,543 and 744, respectively) and randomized these respondents to be interviewed by T-ACASI or telephone interviewer-administered questioning (T-IAQ). Response distributions for all subjective measurements obtained by T-ACASI diverge from those obtained by human telephone interviewers. For six of our seven ordinal-scaled measurements, this divergence involved shifting responses directionally along the ordinal scale, as opposed to a nondirectional redistribution among response categories. When interviewed by T-ACASI, respondents were more supportive of traditional gender roles and corporal punishment, less supportive of integrated neighborhoods and same-gender sex, and more likely to agree that occasional marijuana use is harmless and to describe themselves as attractive. The majority of these results suggest that telephone survey respondents may provide more "tolerant" and "socially liberal" responses to human interviewers than to a T-ACASI computer. Similarly, although the evidence is not entirely consistent, the impact of T-ACASI appears to increase with the social vulnerability of the population surveyed.

Taking the Audio Out of Audio-CASI

Couper, M. P., Tourangeau, R., Marvin, T. — 2009-06-12

Audio computer-assisted self-interviewing (audio-CASI or ACASI) has been widely adopted around the world as a method for eliciting more candid responses to sensitive questions in surveys. While few studies have explored the added advantages audio may bring over text-CASI, those that have (e.g., Tourangeau and Smith, 1996, Public Opinion Quarterly 60(2):275–304) found large effects for self administration over interviewer administration, but only modest additional gains from the audio enhancement. In this paper, we explore the use of audio-CASI versus text-CASI in a national survey of fertility-related issues in the United States (the National Survey of Family Growth). In the pretest, male and female respondents were randomly assigned to audio-CASI (n = 299) or text-CASI (n = 312). We compare the distributions of substantive responses between modes and examine a variety of paradata (e.g., keystroke files, time stamps) to examine the use of the CASI instruments. The main study, which interviewed 7,643 women and 4,928 men aged 15–44 in 2002–03, used audio-CASI only, but again we have a variety of paradata and interviewer debriefing items to examine the extent to which subjects made use of the audio enhancements to CASI. Our results indicate that most respondents make limited use of the audio features of audio-CASI and accordingly the gains produced by this technology are modest relative to text-CASI.

Optimal Design of Branching Questions to Measure Bipolar Constructs

Malhotra, N., Krosnick, J. A., Thomas, R. K. — 2009-06-12

Scholars routinely employ rating scales to measure attitudes and other bipolar constructs via questionnaires, and prior research indicates that this is best done using sequences of branching questions in order to maximize measurement reliability and validity. To identify the optimal design of branching questions, this study analyzed data from several national surveys using various modes of interviewing. We compared two branching techniques and different ways of using responses to build rating scales. Three general conclusions received empirical support: (1) after an initial three-option question assessing direction (e.g., like, dislike, neither), respondents who select one of the endpoints should be asked to choose among three levels of extremity, (2) respondents who initially select a midpoint with a precise label should not be asked whether they lean one way or the other, and (3) bipolar rating scales with seven points yield measurement accuracy superior to that of three-, five-, and nine-point scales.

Open-Ended Questions in Web Surveys: Can Increasing the Size of Answer Boxes and Providing Extra Verbal Instructions Improve Response Quality?

Smyth, J. D., Dillman, D. A., Christian, L. M., Mcbride, M. — 2009-06-12

Previous research has revealed techniques to improve response quality in open-ended questions in both paper and interviewer-administered survey modes. The purpose of this paper is to test the effectiveness of similar techniques in web surveys. Using data from a series of three random sample web surveys of Washington State University undergraduates, we examine the effects of visual and verbal answer-box manipulations (i.e., altering the size of the answer box and including an explanation that answers could exceed the size of the box) and the inclusion of clarifying and motivating introductions in the question stem. We gauge response quality by the amount and type of information contained in responses as well as response time and item nonresponse. The results indicate that increasing the size of the answer box has little effect on early responders to the survey but substantially improved response quality among late responders. Including any sort of explanation or introduction that made response quality and length salient also improved response quality for both early and late responders. In addition to discussing these techniques, we also address the potential of the web survey mode to revitalize the use of open-ended questions in self-administered surveys.

The Framing of the Survey Request and Panel Attrition

Tourangeau, R., Ye, C. — 2009-06-12

Every survey begins with a request to the sample members to take part. How that request is framed can have a variety of consequences, including its intended (positive) effect on the cooperation rate. Survey appeals tend to emphasize the benefits of participation, but there is reason to think that emphasizing the negative consequences of nonparticipation may sometimes be a more effective method of inducing cooperation. We carried out an experiment in which respondents in a random digit dialing (RDD) sample were asked to complete a second telephone interview. For approximately half of the respondents, we emphasized the benefits of their completing the follow-up interview; for the others, we emphasized the loss involved if they chose not to complete the follow-up. Based on Kahneman and Tversky's prospect theory, we predicted that the loss framing would be more effective than the gain framing. In line with our prediction, 87.5 percent of those who got the "loss" framing of the request completed the second interview versus 77.9 percent of those who got the "gain" framing. Multivariate models of the response rate to the second interview (conditional on completion of the first) suggest that the framing effect is fairly robust across subgroups of the sample.

Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey

Galesic, M., Bosnjak, M. — 2009-06-12

This paper investigates how expected and actual questionnaire length affects cooperation rates and a variety of indicators of data quality in web surveys. We hypothesized that the expected length of a web-based questionnaire is negatively related to the initial willingness to participate. Moreover, the serial position of questions was predicted to influence four indicators of data quality. We hypothesized that questions asked later in a web-based questionnaire will, compared to those asked earlier, be associated with (a) shorter response times, (b) higher item-nonresponse rates, (c) shorter answers to open-ended questions, and (d) less variability to items arranged in grids. To test these assumptions, we manipulated the stated length (10, 20, and 30 minutes) and the position of questions in an online questionnaire consisting of randomly ordered blocks of thematically related questions. As expected, the longer the stated length, the fewer respondents started and completed the questionnaire. In addition, answers to questions positioned later in the questionnaire were faster, shorter, and more uniform than answers to questions positioned near the beginning.

Can A Deadline and Compressed Mailing Schedule Improve Mail Response in the Decennial Census?

Martin, E. — 2009-06-12

This study reports results of a national mailout–mailback experiment that evaluated the effects of a deadline and compressed mailing schedule on response to census mail questionnaires. The experiment finds that providing a deadline and shorter interval (by one week) in which to complete the census form leads to a significantly higher rate of response (by 2.0 percentage points). Data quality was not impaired and in some cases showed significant improvements in the experimental panel (lower item nonresponse for one item and slight improvements in coverage). These results contrast with previous studies suggesting that a due date is not effective for improving response in mail surveys, and imply that a deadline would lead to significant cost savings in the decennial census.

Improving Mail Survey Response Rates Using Chocolate and Replacement Questionnaires

Brennan, M., Charbonneau, J. — 2009-06-12

This study examined the effectiveness of two procedures aimed at improving response rates for mail surveys: an incentive sent with either the first mail-out or first follow-up, and the inclusion of a replacement questionnaire, with or without an incentive, with either the first or second follow-up. The survey involved a sample of 1,600 New Zealand residents aged 18 years or older, randomly selected from the 2005 Electoral Roll and randomly assigned to one of four treatment groups. The response rates after two follow-ups ranged from 62.3 to 66.5 percent. Using a chocolate as an incentive with the first mail-out was effective in generating a significantly higher initial response than the control; however, after two follow-up mail-outs, differences were not significant. Sending a replacement questionnaire plus a chocolate with the first follow-up generated a significantly higher response rate than merely sending a letter, and the effect of this procedure persisted through to the end of the survey. Overall, these results provide further compelling evidence of the importance of using follow-up mail-outs for improving mail survey response rates.

The Draft Lottery and Attitudes Towards the Vietnam War

Bergan, D. E. — 2009-06-12

The most striking and theoretically anomalous finding of previous research on self-interest and attitudes is the absence of a self-interest motive in support for the Vietnam War. This research note reconsiders this result using a panel survey of university students collected before and after the first Vietnam draft lottery. These data are unique because they allow the unbiased estimation of the effect of self-interest on attitudes toward the war. I find that, contrary to previous results, self-interest had a substantial impact on support for the war.

Strategies for Surveys of American Indians

Lavelle, B., Larsen, M. D., Gundersen, C. — 2009-06-12

Although past researchers have sometimes labeled American Indians "hard-to-reach," a review of the literature identifies several available strategies for improving sampling and response rates for surveys of this population. After discussing various approaches to defining who is American Indian, this article summarizes the literature on challenges in conducting research in American Indian populations. The literature strongly points to the need for full and active partnerships between American Indian communities and outside researchers in order to ensure that the communities are respected, protected, and benefit from the research. Full and active partnerships also are needed to improve the quality of data collected in American Indian communities. Successful strategies used in U.S. Census Bureau enumeration efforts exemplify the recommendations for participatory research with American Indian communities.

Trends--Tolerance in the United States

Schafer, C. E., Shaw, G. M. — 2009-06-12

Recent survey data illustrate growing tolerance of various traditionally unpopular groups and their rights to teach, speak publicly, live near us as neighbors, and have their books reside in public libraries. Gay and lesbian people perhaps have enjoyed the largest shift in tolerance during the past decade and a half, though other groups have come to enjoy greater acceptance as well. This article presents trend data regarding public attitudes toward feminists, Jews, atheists, persons with strong religious beliefs, immigrants, Muslims, militarists, and AIDS sufferers since 1990. Groups that have recently experienced poorer acceptance include Muslims since 9/11 and immigrants more broadly over the past decade. Most other groups asked about in surveys have come to enjoy more acceptance. Despite the broad patterns of growing tolerance toward people who are unlike most respondents, it remains quite possible that Americans have shifted their intolerance toward other least-liked groups.

David W. Moore. The Opinion Makers: An Insider Exposes the Truth behind the Polls

Traugott, M. W. — 2009-06-12

W. Lance Bennett, Regina G. Lawrence, and Steven Livingston. When the Press Fails: Political Power and the News Media from Iraq to Katrina

Sheafer, T. — 2009-06-12

Editors' Note

Druckman, J. N., Mathiowetz, N. A. — 2009-04-06

Core Networks and Whites' Attitudes Toward Immigrants and Immigration Policy

Berg, J. A. — 2009-04-06

Recent immigration has made the United States significantly more racially and ethnically diverse. These demographic changes prompt questions regarding intergroup conflict. With data from the 2004 General Social Survey and the 2000 Census, I add to this discussion by using multilevel models to evaluate a network perspective in predicting native-born whites’ attitudes toward immigrants and immigration policy. The results indicate that native-born whites who are embedded in educated core networks with nonwhite alters are likely to hold pro-immigrant attitudes, while those who are embedded in older and tighter core networks are likely to hold anti-immigrant attitudes, controlling for individual- and group-level factors. Personal contacts play an important role in shaping native-born whites’ opinions of immigrants and immigration policy, regardless of the presence of or interaction with immigrants. At the same time, core networks also condition the effects of group threat and intergroup contact on immigration attitudes, suggesting that the interpersonal environment and the broader social environment interact during opinion formation.

Sources of Error in Cognitive Interviews

Conrad, F. G., Blair, J. — 2009-04-06

Cognitive interviewing is used to identify problems in questionnaires under development by asking a small number of pretest participants to verbally report their thinking while answering the draft questions. Just as responses in production interviews include measurement error, so the detection of problems in cognitive interviews can include error. In the current study, we examine error in the problem detection of both cognitive interviewers evaluating their own interviews and independent judges listening to the full set of interviews. The cognitive interviewers were instructed to probe for additional information in one of two ways: the Conditional Probe group was instructed to probe only about what respondents had explicitly reported; the Discretionary Probe group was instructed to probe whenever they felt it appropriate. Agreement about problems was surprisingly low overall, but differed by interviewing technique. The Conditional Probe interviewers uncovered fewer potential problems but with higher inter-judge reliability than did the Discretionary Probe interviewers. These differences in reliability were related to the type of probes. When interviewers in either group probed beyond the content of respondents’ verbal reports, they were prone to believe that the respondent had experienced a problem when the majority of judges did not believe this to be the case (false alarms). Despite generally poor performance at the level of individual verbal reports, judges reached relatively consistent conclusions across the interviews about which questions most needed repair. Some practical measures may improve the conclusions drawn from cognitive interviews but the quality of the findings is limited by the content of the verbal reports.

Use of Expert Ratings as Sampling Strata for a More Cost-Effective Probability Sample of a Rare Population

Elliott, M. N., McCaffrey, D., Perlman, J., Marshall, G. N., Hambarsoomians, K. — 2009-04-06

We consider situations in which externally observable characteristics allow experts to quickly categorize individual households as likely or unlikely to contain a member of a rare target population. This classification can form the basis of disproportionate stratified sampling such that households classified as "unlikely" are sampled at a lower rate than those classified as "likely," thereby reducing screening costs. Design weights account for this approach and allow unbiased estimates for the target population. We demonstrate that with sensitivity and specificity of expert classification at least 70 percent, and ideally at least 80 percent, our approach can economically increase effective sample size for a rare population. We develop heuristics for implementing this approach and demonstrate that sensitivity drives design effects and screening costs whereas specificity only drives the latter. We demonstrate that the potential gains from this approach increase as the target population becomes rarer. We further show that for most applications, unlikely strata should be sampled at 1/6 to 1/2 the rate of likely strata. This approach was applied to a survey of Cambodian immigrants in which the 82 percent of households rated "unlikely" were sampled at 1/4 the rate as "likely" households, reducing screening from 9.4 to 4.0 approaches per complete. Sensitivity and specificity were 86 percent and 91 percent, respectively. Weighted estimation had a design effect of 1.26, so screening costs per effective sample size were reduced by 47 percent. We also note that in this instance, expert classification appeared to be uncorrelated with survey outcomes of interest among eligibles.

Survey Breakoff

Peytchev, A. — 2009-04-06

Survey respondents may start the survey but fail to complete it, resulting in breakoff. This behavior occurs in interviewer-administered surveys but it occurs at high rates in web surveys, necessitating further attention. Breakoff is one of multiple response behaviors, each of which can be affected by unique and common causes. We present a framework within which to study different response behaviors, unit nonresponse, breakoff, item nonresponse, and measurement properties, and the factors affecting them in web surveys. Theories within this framework for explaining breakoff are proposed and tested. Features within the survey that are only seen after starting are predictive of breakoff rate, distinguishing this behavior from unit nonresponse. Education, a proxy for respondent cognitive sophistication, was significantly related to breakoff rates. Furthermore, respondents who broke off did not seem inattentive, supporting further efforts in their retention. Question and questionnaire design characteristics were predictive of survey breakoff, and placed in the context of the survey response process model and respondent burden. Implications for survey design and needs for future work are discussed.

Political Engagement, Mobilization, and Direct Democracy

Donovan, T., Tolbert, C. J., Smith, D. A. — 2009-04-06

Research has found that states using initiatives and referendums have higher turnout, particularly in midterm elections. Existing research has not examined who is mobilized to vote when issues appear on statewide ballots. Building on work by Campbell (1966. "Surge and Decline: A Study of Electoral Change." In Elections and the Political Order, eds. A. Campbell, P. E. Converse, W. E. Miller, and D. E. Stokes. New York: Wiley), we test whether ballot measures engage and mobilize people who do not fit the profile of regular voters. Using national opinion data from the 2004 and 2006 elections, we find that independents (relative to partisans) exhibited greater awareness of and interest in ballot measures in the midterm election. This pattern is not found in the presidential election, where peripheral voters are likely to be mobilized by the stimulus of the presidential race rather than by ballot measures. Absent salient ballot measures, some episodic independent voters may not be engaged by midterm elections. This suggests that some variation in midterm turnout maybe a function of peripheral voters becoming engaged by ballot measures.

Campaign Dynamics in Battleground and Nonbattleground States

Panagopoulos, C. — 2009-04-06

Strategic considerations rooted in the Electoral College system of presidential elections compel presidential campaigns to allocate resources disproportionately to competitive states. In contemporary campaigns, this results in a situation in which many voters are entirely unexposed to campaign activities over the duration of the campaign. The implications of lopsided communications that relegate voters in uncompetitive states to bystander status in presidential campaigns are potentially significant and merit greater scrutiny. In this study, I analyze how the dynamics of voter preferences in uncompetitive states compare to those in battleground states. Using data from the 2000 National Annenberg Election Survey, I find that the preferences of voters in battleground states are more variable but more stable than their counterparts in nonbattleground states. As expected, I also find stronger evidence of campaign effects in battleground states.

The Immensely Inflated News Audience: Assessing Bias in Self-Reported News Exposure

Prior, M. — 2009-04-06

Many studies of media effects use self-reported news exposure as their key independent variable without establishing its validity. Motivated by anecdotal evidence that people's reports of their own media use can differ considerably from independent assessments, this study examines systematically the accuracy of survey-based self-reports of news exposure. I compare survey estimates to Nielsen estimates, which do not rely on self-reports. Results show severe overreporting of news exposure. Survey estimates of network news exposure follow trends in Nielsen ratings relatively well, but exaggerate exposure by a factor of 3 on average and as much as eightfold for some demographics. It follows that apparent media effects may arise not because of differences in exposure, but because of unknown differences in the accuracy of reporting exposure.

Political Trust, Ideology, and Public Support for Tax Cuts

Rudolph, T. J. — 2009-04-06

This article analyzes the relationship between political trust, ideology, and public support for tax cutting. Conceptualizing tax cuts as government action under conditions of risk, I argue that political trust should increase rather than decrease public support for tax cutting, particularly among those for whom endorsement of tax cuts entails ideological sacrifice. The results show that political trust increases support for a variety of tax cut initiatives, but only among liberals. The results further suggest that trust is an instrumental resource that can bolster support for a conservative as well as a liberal policy agenda.

Pre-Election Polling: Identifying Likely Voters Using Iterative Expert Data Mining

Murray, G. R., Riley, C., Scime, A. — 2009-04-06

One often-noted difficulty in pre-election polling is the identification of likely voters. Our objective is to build a likely voter model for presidential elections that efficiently balances accuracy and number of questions used. We employ the Iterative Expert Data Mining technique and data from the American National Election Studies to identify a small number of survey questions that can be used to classify likely voters while maintaining or surpassing the accuracy rates of other models. Specifically, we propose two survey items that together correctly classify 78 percent of respondents as voters or nonvoters over a multielection, multidecade period. We argue that our proposed model compares favorably to competing models by capturing the successful elements of those models while ignoring other elements that constrain identification. We end by suggesting that our model offers a new approach to identifying and evaluating likely voters that may maintain or increase accuracy without also increasing cost.

Context Effects and Social Change

Schuman, H. — 2009-04-06

Strack and Martin (1987) have proposed that although probability samples of general populations are needed for studying attitude content, psychological processes such as context and other response effects can be investigated quite well or even better with homogeneous convenience samples. The present paper shows that, in contrast, when social change is included as an essential variable in a study of a well-known context effect, the process itself is weakened and may even disappear. The distinction between content and process is not always tenable in surveys once we include long-term social change as a variable, as we must as the horizon for survey research extends further and further into the future.

Order Effects in Complex and Simple Tasks

Malhotra, N. — 2009-04-06

There is strong evidence that the order in which response options are presented in surveys significantly affects the answers that respondents provide. According to the theory of survey satisficing, the severity of order effects should increase with task difficulty. However, the tasks provided to respondents in existing studies of response-order effects are generally very simple, making it difficult to evaluate the satisficing hypothesis. Further, evidence from cognitive psychology suggests a completely different mechanism: people are more motivated to persist in completing tasks when they are intricate, challenging, and enriching. I designed survey experiments administered over the Internet consisting of two types of tasks: (1) a complex task in which respondents were asked to rank seven public officials in order of how much they should be blamed for the property damage and loss of life caused by Hurricane Katrina in the city of New Orleans; and (2) a series of simple tasks in which respondents answered items with ordinal response choices on rating scales. I found almost no order effects in the complex task among all educational groups. Conversely, I found significant and substantial order effects in the simple tasks, particularly among low-education respondents. These results suggest that theories of survey satisficing may simplify matters by assuming that satisficing monotonically increases with task difficulty. Moreover, my findings have important implications for questionnaire design, underscoring the importance of randomizing response options.

The Polls--Trends Confidence in Law Enforcement

Shaw, G. M., Brannan, K. E. — 2009-04-06

Public confidence in law enforcement remains high but showed some signs of slight decline from the late 1990s to the middle of this decade. Confidence in law enforcement fluctuated some in apparent response to the terrorist attacks of September 11 and has not been particularly strong when viewed through a racial lens. In keeping with falling crime rates through the first part of this decade followed by a recent rise, the percentage of the public that believed anticrime spending to be too little declined slightly from the late 1990s to 2005 but rebounded in 2006.

Andrew Gelman, David Park, Boris Shor, Joseph Bafumi, and Jeronimo Cortina. Red State, Blue State, Rich State, Poor State: Why Americans Vote the Way They Do

Lenz, G. S. — 2009-04-06

D. Sunshine Hillygus and Todd G. Shields. The Persuadable Voter: Wedge Issues in Presidential Campaigns

Sides, J. — 2009-04-06

Larry M. Bartels. Unequal Democracy: The Political Economy of the New Gilded Age

Winter, N. J. G. — 2009-04-06

Manuscript Referees, 2007-2008

2009-04-06

Web Survey Methods: Introduction

Couper, M. P., Miller, P. V. — 2009-02-11

Face-to-Face versus Web Surveying in a High-Internet-Coverage Population: Differences in Response Quality

Heerwegh, D., Loosveldt, G. — 2009-02-11

The current study experimentally investigates the differences in data quality between a face-to-face and a web survey. Based on satisficing theory, it was hypothesized that web survey respondents would be more likely to satisfice for a multitude of reasons, thereby producing data of lower quality. The data show support for the hypothesis. Web survey respondents were shown to produce a higher "don't know" response rate, to differentiate less on rating scales, and to produce more item nonresponse than face-to-face survey respondents.

Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity

Kreuter, F., Presser, S., Tourangeau, R. — 2009-02-11

Although it is well established that self-administered questionnaires tend to yield fewer reports in the socially desirable direction than do interviewer-administered questionnaires, less is known about whether different modes of self-administration vary in their effects on socially desirable responding. In addition, most mode comparison studies lack validation data and thus cannot separate the effects of differential nonresponse bias from the effects of differences in measurement error. This paper uses survey and record data to examine mode effects on the reporting of potentially sensitive information by a sample of recent university graduates. Respondents were randomly assigned to one of three modes of data collection—conventional computer-assisted telephone interviewing (CATI), interactive voice recognition (IVR), and the Web—and were asked about both desirable and undesirable attributes of their academic experiences. University records were used to evaluate the accuracy of the answers and to examine differences in nonresponse bias by mode. Web administration increased the level of reporting of sensitive information and reporting accuracy relative to conventional CATI, with IVR intermediate between the other two modes. Both mode of data collection and the actual status of the respondent influenced whether respondents found an item sensitive.

Eliciting Subjective Probabilities in Internet Surveys

Delavande, A., Rohwedder, S. — 2009-02-11

Individuals’ subjective expectations are important in explaining heterogeneity in individual choices, but their elicitation poses some challenges, in particular when one is interested in the subjective probability distribution of an individual. We have developed an innovative visual representation for Internet surveys that has some advantages over previously used formats. In this paper we present our findings from testing this visual representation in the context of individuals’ Social Security expectations. Respondents are asked to allocate a total of 20 balls across seven bins to express what they believe the chances to be that their future Social Security benefits would fall into any one of those bins. Our data come from the Internet survey of respondents to the Health and Retirement Study, a representative survey of the U.S. population aged 51 and older. To contrast the results from the visual format with a previously used format, we divided the sample into two random groups and administered both, the visual format and the more standard percent chance format. Our findings suggest that the main advantage of the visual format is that it generates usable answers for virtually all respondents in the sample while in the percent chance format a significant fraction (about 20 percent) of responses is lost due to inconsistencies. Across various other dimensions, the visual format performs similarly to the percent chance format, leading us to conclude that the bins-and-balls format is a viable alternative that leads to more complete data.

Eye-Tracking Data: New Insights on Response Order Effects and Other Cognitive Shortcuts in Survey Responding

Galesic, M., Tourangeau, R., Couper, M. P., Conrad, F. G. — 2009-02-11

Survey researchers since Cannell have worried that respondents may take various shortcuts to reduce the effort needed to complete a survey. The evidence for such shortcuts is often indirect. For instance, preferences for earlier versus later response options have been interpreted as evidence that respondents do not read beyond the first few options. This is really only a hypothesis, however, that is not supported by direct evidence regarding the allocation of respondent attention. In the current study, we used a new method to more directly observe what respondents do and do not look at by recording their eye movements while they answered questions in a Web survey. The eye-tracking data indicate that respondents do in fact spend more time looking at the first few options in a list of response options than those at the end of the list; this helps explain their tendency to select the options presented first regardless of their content. In addition, the eye-tracking data reveal that respondents are reluctant to invest effort in reading definitions of survey concepts that are only a mouse click away or paying attention to initially hidden response options. It is clear from the eye-tracking data that some respondents are more prone to these and other cognitive shortcuts than others, providing relatively direct evidence for what had been suspected based on more conventional measures.

Completion Time and Response Order Effects in Web Surveys

Malhotra, N. — 2009-02-11

The use of the World Wide Web to conduct surveys has grown rapidly over the past decade, raising concerns regarding data quality, questionnaire design, and sample representativeness. This research note focuses on an issue that has not yet been studied: Are respondents who complete self-administered Web surveys more quickly—perhaps taking advantage of participation benefits while minimizing effort—also more prone to response order effects, a manifestation of "satisficing"? I surveyed a random sample of the US adult population over the Web and manipulated the order in which respondents saw the response options. I then assessed whether primacy effects were moderated by the overall length of time respondents took to complete the questionnaires. I found that low-education respondents who filled out the questionnaire most quickly were most prone to primacy effects when completing items with unipolar rating scales. These results have important implications for various aspects of Web survey methodology including panel management, human–computer interaction, and response order randomization.

Facial Similarity between Voters and Candidates Causes Influence

Bailenson, J. N., Iyengar, S., Yee, N., Collins, N. A. — 2009-02-11

Social science research demonstrates that people are drawn to others perceived as similar. We extend this finding to political candidates by comparing the relative effects of candidate familiarity as well as partisan, issue, gender, and facial similarity on voters’ evaluations of candidates. In Experiment 1, during the week of the 2006 Florida gubernatorial race, a national representative sample of voters viewed images of two unfamiliar candidates (Crist and Davis) morphed with either themselves or other voters. Results demonstrated a strong preference for facially similar candidates, despite no conscious awareness of the similarity manipulation. In Experiment 2, one week before the 2004 presidential election, a national representative sample of voters evaluated familiar candidates (Bush and Kerry). Strong partisans were unmoved by the facial similarity manipulation, but weak partisans and independents preferred the candidate with whom their own face had been morphed over the candidate morphed with another voter. In Experiment 3, we compared the effects of policy similarity and facial similarity using a set of prospective 2008 presidential candidates. Even though the effects of party and policy similarity dominated, facial similarity proved a significant cue for unfamiliar candidates. Thus, the evidence across the three studies suggests that even in high-profile elections, voters prefer candidates high in facial similarity, but most strongly with unfamiliar candidates.

Does a Probability-Based Household Panel Benefit from Assignment to Postal Response as an Alternative to Internet-Only?

Rookey, B. D., Hanway, S., Dillman, D. A. — 2009-02-11

A potential limitation of web-only panels of the general public, even when households are selected using probability methods, is that only about 70 percent of U.S. households have members with Internet access. In addition, some members of Internet-connected households may be unable or unwilling to participate over the web. The Gallup Panel uses both web and mail modes to survey respondents and in 2006 included approximately 50,000 households selected by random-digit dialing. Frequent Internet users were assigned to respond by the web, while others were assigned to participate by mail using a paper questionnaire with a similar visual layout to the web. We use several approaches to determine whether or not the mail option adds value to the results in an otherwise Internet panel and organize our analyses around answering a series of questions. First, does the use of mail allow different types of people to be included? Second, do mail and web respondents give different answers to the same questions? Third, does weighting on and controlling for demographics eliminate any differences in responses from mail and web respondents and indicate that mail is not needed? Finally, do differences exist when responses are collected using an independent mode? In general, the answers to these questions suggest that use of mail adds value to the panel results and improves the overall accuracy of survey results.

Effects of Design in Web Surveys: Comparing Trained and Fresh Respondents

Toepoel, V., Das, M., Van Soest, A. — 2009-02-11

In this paper, we investigate whether there are differences in the effect of instrument design between trained and fresh respondents. In three experiments, we varied the number of items on a screen, the choice of response categories, and the layout of a five-point rating scale. In general, effects of design carry over between trained and fresh respondents. We found little evidence that survey experience influences the question-answering process. Trained respondents seem to be more sensitive to satisficing. The shorter completion time, higher interitem correlations for multiple-item-per-screen formats, and the fact that they select the first response options more often indicate that trained respondents tend to take shortcuts in the response process and study the questions less carefully.

Computing Response Metrics for Online Panels

Callegaro, M., DiSogra, C. — 2009-02-11

As more researchers use online panels for studies, the need for standardized rates to evaluate these studies becomes paramount. There are currently many different ways and conflicting terminology used to compute various metrics for online panels. This paper discusses the sparse literature on how to compute response, refusal, and other rates and proposes a set of formulas and a standardized terminology that can be used to calculate and interpret these metrics for online panel studies. A description of and distinction between probability-based and volunteer opt-in panels is made since not all metrics apply to both types. A review of the existing discussion and recommendations, mostly from international organizations, is presented for background and context. In order to propose response and other metrics, the different stages involved in building an online panel are delineated. Metrics associated with these stages contribute to cumulative response rate formulas that can be used to evaluate studies using online probability-based panels. (Only completion rates can be calculated with opt-in panels.) We conclude with a discussion of the meaning of the different metrics proposed and what we think should be reported for which type of panel.

Public Opinion Quarterly - recent issues

Subscriptions

Editorial Board

Contents

Predictors of Public Support for Gender-Related Affirmative Action: Interests, Gender Attitudes, and Stratification Beliefs

Impact of T-ACASI on Survey Measurements of Subjective Phenomena

Taking the Audio Out of Audio-CASI

Optimal Design of Branching Questions to Measure Bipolar Constructs

Open-Ended Questions in Web Surveys: Can Increasing the Size of Answer Boxes and Providing Extra Verbal Instructions Improve Response Quality?

The Framing of the Survey Request and Panel Attrition

Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey

Can A Deadline and Compressed Mailing Schedule Improve Mail Response in the Decennial Census?

Improving Mail Survey Response Rates Using Chocolate and Replacement Questionnaires

The Draft Lottery and Attitudes Towards the Vietnam War

Strategies for Surveys of American Indians

Trends--Tolerance in the United States

David W. Moore. The Opinion Makers: An Insider Exposes the Truth behind the Polls

W. Lance Bennett, Regina G. Lawrence, and Steven Livingston. When the Press Fails: Political Power and the News Media from Iraq to Katrina

Editors' Note

Core Networks and Whites' Attitudes Toward Immigrants and Immigration Policy

Sources of Error in Cognitive Interviews

Use of Expert Ratings as Sampling Strata for a More Cost-Effective Probability Sample of a Rare Population

Survey Breakoff

Political Engagement, Mobilization, and Direct Democracy

Campaign Dynamics in Battleground and Nonbattleground States

The Immensely Inflated News Audience: Assessing Bias in Self-Reported News Exposure

Political Trust, Ideology, and Public Support for Tax Cuts

Pre-Election Polling: Identifying Likely Voters Using Iterative Expert Data Mining

Context Effects and Social Change

Order Effects in Complex and Simple Tasks

The Polls--Trends Confidence in Law Enforcement

Andrew Gelman, David Park, Boris Shor, Joseph Bafumi, and Jeronimo Cortina. Red State, Blue State, Rich State, Poor State: Why Americans Vote the Way They Do

D. Sunshine Hillygus and Todd G. Shields. The Persuadable Voter: Wedge Issues in Presidential Campaigns

Larry M. Bartels. Unequal Democracy: The Political Economy of the New Gilded Age

Manuscript Referees, 2007-2008

Web Survey Methods: Introduction

Face-to-Face versus Web Surveying in a High-Internet-Coverage Population: Differences in Response Quality

Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity

Eliciting Subjective Probabilities in Internet Surveys

Eye-Tracking Data: New Insights on Response Order Effects and Other Cognitive Shortcuts in Survey Responding

Completion Time and Response Order Effects in Web Surveys

Facial Similarity between Voters and Candidates Causes Influence

Does a Probability-Based Household Panel Benefit from Assignment to Postal Response as an Alternative to Internet-Only?

Effects of Design in Web Surveys: Comparing Trained and Fresh Respondents

Computing Response Metrics for Online Panels