Using AI to improve our understanding of verbal confidence and to aid decision-making: Eyewitness lineup identification as a model case

From medical treatment to predicting recidivism, artificial intelligence (AI) is playing an increasingly larger role in human decision-making – in large part because AI predictions are superior to human predictions in a variety of different domains. This project uses eyewitness lineup identifications as a model paradigm. One aim of the project uses AI to better understand verbal expressions of confidence about an identification. For example, when an eyewitness states “I’m pretty sure it’s him” about a lineup identification what is the likelihood that the eyewitness’s identification is correct? Another aim of the project examines how best to convey AI output to people so as to improve their predictions about the accuracy of an eyewitness’s identification. The project consists of two sets of experiments. One set uses variations on an eyewitness memory paradigm to examine the predictive value of verbal (e.g., “I’m pretty certain”) and numeric (e.g., “I’m 75% certain”) expressions of confidence. It is largely assumed that verbal confidence statements reflect the same underlying information as numeric confidence ratings. Machine-learning classifiers are used to quantify verbal confidence to explain why verbal confidence is not redundant with numeric confidence but can contribute unique added value in predicting the accuracy of a response. A second set of experiments test two predictions. First, that machine learning estimates about the accuracy of a lineup identification are more resistant than human estimates to the effects of various kinds of contextual information. Second, that a cognitive-forcing method of conveying machine-learning estimates about eyewitness identification accuracy is most effective at improving people’s predictions about eyewitness accuracy, particularly under conditions when people’s intuitions about eyewitness performance are wrong. Overall, this project addresses a need to identify issues in human-algorithm interactions before the spread of AI-assistance to the domain of eyewitness identification. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.