Collaborative Research: Predictive processing in naturalistic language comprehension through EEG and computational modeling

Humans understand language rapidly and with remarkable ease. One way that humans do this is by predicting what a conversation partner might say next. But, there remain many unanswered questions about how different language backgrounds might help, or hinder, effective predictions. There is a critical need to understand proficient second-language comprehension. This project studies brain signals while bilinguals listen to an audiobook story in their first and second language. Computational modeling using artificial intelligence (AI) language systems are used to test the kinds of predictions people make, the information that guides those predictions, and how predictions are affected by differences in language background. This project offers insight into how AI can incorporate multiple languages in a realistic way and increases awareness of bilingual language with programs targeting future teachers and the public. Other benefits to society include increased transparency and reproducibility in language research by providing a large corpus of brain and behavioral data for other scientists and engineers. To meet these aims, the project collects electroencephalography (EEG) signals from three groups of bilingual participants with different levels of experience while they listen to an audiobook story. These signals reflect fast-changing brain responses and are highly sensitive to expectations in language. AI is used to capture the linguistic features of the story, such as the relationships between nouns and verbs in a sentence and the predictability of upcoming words. Statistical analyses are used to test the alignment between these features and brain activity, showing which features best capture brain activity and how this may be different for different language backgrounds. By training computational models with different amounts of exposure to one or more languages, the project further tests how the statistical properties of different languages modulate brain responses of multilingual language users. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.