CAREER: Towards Daily Immersive Sign Language Training Using Intelligent Wearables and Large Language Model

Many Deaf and hard-of-hearing (DHH) individuals and their families use American Sign Language (ASL). Yet, many struggle with understanding and learning how to use it. This may be due to a lack of trained teachers, school programs, or helpful learning materials. You can find many online resources to learn ASL, but they often lack real-time feedback and portability. The investigator will tackle these challenges by developing an innovative ASL learning platform. This project will lead to a technology that improves the learning of ASL for various users. The investigator will involve the DHH community in the project. This way, the platform will be useful for them and a wider community. To build this platform, this project will meet the following objectives. First, the investigator will design wearable devices integrated with artificial intelligence (AI). This will allow continuous and accurate tracking of facial expressions and hand gestures. Second, the investigator will develop advanced sensing technologies and AI algorithms. This will ensure the platform works with limited data and in different contexts. Third, the system will offer real-time feedback to users. This will help them to learn ASL with the help of a Large Language Model (LLM). This research will advance ASL learning, behavior recognition, wearable technology, and artificial intelligence. It will also cover signal processing, deep learning, and large language model applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.