EAGER: Nonintrusive Engagement and Posture Detection in Virtual Classroom Environments

Virtual classes and other online engagements continue to remain popular in the post-pandemic world because of their convenience and time-saving potential. Unfortunately, a major drawback of virtual attendance is the difficulty in assessing participant engagement, and hence, difficulty in tailoring content delivery and addressing specific participant issues. The purpose of this EAGER project is to explore whether nonintrusive imaging and audio-based monitoring can reliably gauge physical and physiological parameters relevant to engagement assessment. Such an assessment can be done privately for each participant and can form the basis for discreet feedback, both to the participant and the teacher/leader. This involves challenges associated with the accurate, online analysis of various measures, such as facial expressions, eye gaze, posture, body movements, and physiological attributes (e.g., heart rate, breathing rate). Further challenges include addressing individual variabilities and inherent “noise” (i.e., changes in physical and physiological measures unrelated to the online participation). Thus, the key intellectual merit of the project is in exploring novel combinations of deep learning and logic reasoning-based methods for estimating and combining (fusing) these measures to establish how accurately and consistently remote monitoring can be used to assess engagement and help improve virtual learning environments. The key broader impact of the project lies in its potential to make online learning environments richer and more productive, perhaps even exceeding the limits of direct human perception in nonvirtual settings, while preserving the inherent advantages of virtual participation. If successful, the explored methods have the potential to transform the dynamics of teacher-student/leader-participant interactions in virtual learning/meeting environments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.