Postdoctoral Fellowship: SPRF: Documenting Patterns of Variation in Sign Language Using Computer Vision

Under the sponsorship of Dr. Naomi Caselli at Boston University, this postdoctoral fellowship award supports an early career scientist using automated computer vision techniques to document language production patterns in the visuo-spatial modality. This project extracts measurements of movement features such as speed, space, and energy from video data and compares patterns between different groups of signers. In the process, the researcher will develop applications of new machine learning tools. The resources invested in this work lay the groundwork for efficient and accurate automated gesture and visual language technology by facilitating accelerated development of tools like virtual assistants and communication applications that rely on visual data processing. The project also generates evidence-based findings to inform interpreter training. By enabling more accurate and cost-effective communication during emergencies or other high-stakes events, as well as by identifying ways for hearing interpreters to avoid work-related injuries, this work’s findings also enhance public safety. Using cutting-edge computer vision and machine learning techniques, this research develops methods for capturing and quantifying subtle variation in visual language production. The study applies automated annotation frameworks using 2D video-based body pose estimation to extract detailed measurements of velocity, signing space, and kinetic energy. By comparing these metrics across different signing populations, the project answers questions about how sign language production varies across signers. The development of scalable, automated annotation methods solves longstanding bottlenecks that have stagnated progress in both linguistic research and the engineering of language recognition and communication technologies. These theoretical and technical advances will ultimately lead to greater public safety and workplace efficiency by facilitating accurate and efficient communication in emergencies, healthcare settings, and other high-stakes contexts. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.