Public Opinion Quarterly Advance Access published online on March 19, 2009
Public Opinion Quarterly, doi:10.1093/poq/nfp008
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
Order Effects in Complex and Simple Tasks
Address correspondence to Neil Malhotra; e-mail: neilm{at}stanford.edu.
There is strong evidence that the order in which response options are presented in surveys significantly affects the answers that respondents provide. According to the theory of survey satisficing, the severity of order effects should increase with task difficulty. However, the tasks provided to respondents in existing studies of response-order effects are generally very simple, making it difficult to evaluate the satisficing hypothesis. Further, evidence from cognitive psychology suggests a completely different mechanism: people are more motivated to persist in completing tasks when they are intricate, challenging, and enriching. I designed survey experiments administered over the Internet consisting of two types of tasks: (1) a complex task in which respondents were asked to rank seven public officials in order of how much they should be blamed for the property damage and loss of life caused by Hurricane Katrina in the city of New Orleans; and (2) a series of simple tasks in which respondents answered items with ordinal response choices on rating scales. I found almost no order effects in the complex task among all educational groups. Conversely, I found significant and substantial order effects in the simple tasks, particularly among low-education respondents. These results suggest that theories of survey satisficing may simplify matters by assuming that satisficing monotonically increases with task difficulty. Moreover, my findings have important implications for questionnaire design, underscoring the importance of randomizing response options.
NEIL MALHOTRA is with the Graduate School of Business, Stanford University, 518 Memorial Way, Stanford, CA 94305, USA. I gratefully acknowledge Jon Krosnick, Paul Sniderman, Jed Stiglitz, Daniel Schneider, Phil Garland, John Bullock, Alex Kuo, and the anonymous reviewers for valuable suggestions. The data collection was funded by Timeshare Experiments in the Social Sciences (TESS), NSF Grant 0094964, awarded to Diana Mutz and Arthur Lupia. I thank Diana Mutz of TESS and Poom Nukulkij of Knowledge Networks for their time and assistance.