Cheating in Online Surveys: Implications and Solutions

The ease of looking up the answers to questions designed to test factual knowledge threatens measures of knowledge and beliefs in all areas of research in the social and natural sciences. This research project will advance methods of detecting and deterring respondents from looking up the answers ("cheating") in online surveys. It will introduce new methods of assessing how cheating affects measures of knowledge and knowledge disparities between social groups. These methods will be tested across a range of substantive areas, including political knowledge, scientific knowledge, health literacy, and beliefs about social and economic conditions. Depending on the prevalence of cheating, the availability and accuracy of information, and respondents' motivations for cheating, the optimal approach to dealing with cheating may vary. The results of this research will help researchers think about whether cheating is likely to be a problem in their substantive area. More generally, the research will enable more accurate measures of knowledge, beliefs, and cognitive traits. The project will produce a series of online resources and tutorials, open access articles, and an open access book. Code and data generated from the project will be publicly available. This research project will produce a broadly applicable set of methodological tools for dealing with cheating in online surveys, as well as a theoretical framework that helps researchers' reason about whether cheating is likely to be a significant threat in their substantive area. First, the project will identify high-performing strategies for detection and deterrence that can be widely implemented without compromising privacy. Detection methods include indicators of page-switching, question timers, and specialized "catch" questions that are unlikely to be answered correctly without cheating. Deterrence methods include instructions and pledges not to look up the answers. Based on a classification framework, the performance of different implementations of the individual methods will be compared, as well as the benefits of combining multiple strategies. Second, the project will conduct a needs assessment to examine how the prevalence of cheating varies across substantive topics and how this affects the optimal combination of strategies for each topic. Finally, the project will introduce new methods of estimating the bias that cheating introduces into measures of knowledge and knowledge disparities between social groups. The theoretical foundation for these tools will shed light on the factors that determine how cheating affects (or does not affect) measured knowledge, including the availability and accuracy of information about the topic, as well as the extent to which lookups are motivated by ignorance or to confirm what one already knows. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.