Listening Effort and Gaze Strategies During Audiovisual Speech Perception

PROJECT SUMMARY Listening effort and fatigue have serious impacts on quality of life for individuals with hearing loss. As the clinical importance of listening effort becomes more widely acknowledged, there is an increased urgency to improve our understanding of effort and fatigue in contexts relevant to real-world communication. One factor that has the potential to alleviate effort and fatigue in everyday listening is the ability to see a conversation partner’s face. While visual speech cues are known to improve speech intelligibility — particularly for listeners with hearing loss — the link between visual speech and listening effort has not been established. Translating the perceptual benefits of seeing a talker’s face to real listening further hinges on moving beyond video-based, non-interactive contexts and investigating how listeners use visual speech cues during social communication. The goals of this project are twofold. First, we will determine how and when seeing a talker’s face alleviates effort and fatigue for listeners with cochlear implants (CIs), focusing on cognitively and perceptually demanding listening scenarios. Second, we will examine eye gaze behavior during live conversations: how social interaction changes where on the face listeners tend to look, how these patterns differ for listeners with CIs (who likely rely more on visual cues from the talker’s mouth), and how individual differences in gaze behavior relate to the effort of speech perception during conversation. Results will improve our understanding of audiovisual speech perception and effort during natural communication, laying the groundwork for future studies on how listeners with CIs use visual speech to improve the perception of prosody and track multi-talker conversations. This work will inform the development of diagnostic testing to assess multisensory speech perception, as well as rehabilitation strategies to maximize audiovisual benefits for individuals with hearing and/or vision loss.