Question Understanding Aid (QUAID)
A Web Facility that Tests Question Comprehensibility
ARTHUR C. GRAESSER is a professor in the University of Memphis Department of Psychology. ZHIQIANG CAI is a research scientist in the University of Memphis Department of Psychology. MAX M. LOUWERSE is a professor in the University of Memphis Department of Psychology. FRANCES DANIEL is a doctoral student in the University of Illinois, Chicago, Department of Psychology. This research was funded by grants awarded to Graesser by the Statistical Research Division of the U.S. Census Bureau (1998–99, 43-YA-BC-802930), the National Science Foundation (NSF; SES 9977969), and the Office of Naval Research (ONR; N00014-00-1-0917). Any opinions, findings, and conclusions or recommendations expressed in this material are ours and do not necessarily reflect the views of NSF, ONR, or the U.S. Census Bureau.
Address correspondence to Arthur C. Graesser; e-mail: a-graesser{at}memphis.edu.
When respondents do not understand the meaning of a survey question, they will not supply valid and reliable answers. Survey methodologists should therefore benefit from computer tools and other analytical schemes that help them identify problems with questions with respect to comprehension difficulty. We developed a Web facility called Question Understanding Aid (QUAID; www.psyc.memphis.edu/quaid.html) that assists survey methodologists in identifying problems with the wording, syntax, and semantics of questions on questionnaires. The survey methodologist enters the question into the Web facility, along with any context information and answer alternatives that accompany the question. QUAID quickly returns a list of potential problems with question comprehension, including unfamiliar technical terms, vague or imprecise relative terms, vague or ambiguous noun phrases, complex syntax, and working memory overload. This article describes QUAID and some empirical studies that have assessed the validity and utility of QUAID’s critiques of questions. The output of QUAID was compared with the judgments of experts in language, discourse, and cognition during the development of the tool. In one evaluation, expert survey methodologists critiqued and revised problematic questions, whereas in a second evaluation survey methodologists evaluated the quality of original problematic questions, questions revised with the assistance of QUAID, and questions revised without QUAID. In a third evaluation, eye-tracking data were collected while respondents read questions on a computer screen and answered questions aloud. Respondents had a tendency to give up processing difficult questions too early (called an early exit), which potentially threatens the validity of the respondents’ answers. Survey methodologists are encouraged to use QUAID and further evaluate its validity and utility.