Public Opinion Quarterly Advance Access originally published online on April 2, 2009
Public Opinion Quarterly 2009 73(1):32-55; doi:10.1093/poq/nfp013
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Sources of Error in Cognitive Interviews
Address correspondence to Frederick Conrad; e-mail: fconrad{at}isr.umich.edu.
Cognitive interviewing is used to identify problems in questionnaires under development by asking a small number of pretest participants to verbally report their thinking while answering the draft questions. Just as responses in production interviews include measurement error, so the detection of problems in cognitive interviews can include error. In the current study, we examine error in the problem detection of both cognitive interviewers evaluating their own interviews and independent judges listening to the full set of interviews. The cognitive interviewers were instructed to probe for additional information in one of two ways: the Conditional Probe group was instructed to probe only about what respondents had explicitly reported; the Discretionary Probe group was instructed to probe whenever they felt it appropriate. Agreement about problems was surprisingly low overall, but differed by interviewing technique. The Conditional Probe interviewers uncovered fewer potential problems but with higher inter-judge reliability than did the Discretionary Probe interviewers. These differences in reliability were related to the type of probes. When interviewers in either group probed beyond the content of respondents’ verbal reports, they were prone to believe that the respondent had experienced a problem when the majority of judges did not believe this to be the case (false alarms). Despite generally poor performance at the level of individual verbal reports, judges reached relatively consistent conclusions across the interviews about which questions most needed repair. Some practical measures may improve the conclusions drawn from cognitive interviews but the quality of the findings is limited by the content of the verbal reports.
FREDERICK G. CONRAD is with the University of Michigan's Institute for Social Research, 426 Thompson Street, Ann Arbor, MI 48104, USA. JOHNNY BLAIR is with Abt Associates, 4550 Montgomery Avenue, Suite 800 North, Bethesda, MD 20814-3343, USA. The authors would like to thank the following people for their help on this project: Greg Claxton, Nadra Garas, Jane Joseph, Rachel Levenstein, Nileeni Meegama, Nick Prieur, Elena Tracy, and Sara Woldehana. In addition, the authors are grateful to Gordon Willis, Edward Blair, and two anonymous reviewers for valuable comments on earlier drafts of this article. The authors thank the Bureau of Labor Statistics for supporting several of the reported activities through a contract with Westat, Inc. and a subcontract with the University of Maryland. None of the opinions expressed here are those of the Bureau of Labor Statistics; they are those of the authors alone.