Collaborative Research: CPS: Medium: Designing a Closed-loop Unvoiced Interaction Paradigm

Speech is the most natural way for people to communicate with technology. Voice-based interfaces allow users to interact with their phones and other devices without using their hands. However, speaking aloud is not always practical. Voice interfaces struggle to function in noisy environments, can compromise user privacy in public spaces, and may be impractical in quiet settings where it is desirable to avoid disturbing others (for example, libraries, classrooms, hospitals). This project proposes a novel approach that will allow users to communicate silently with their devices by articulating words without producing sound. The proposed system will use sensors incorporated into a socially acceptable ear-worn device to detect jaw movement and small facial vibrations associated with speech. By transforming these signals into words and commands, the system could enable private, hands-free, and unobtrusive interaction in environments where conventional voice interfaces are ineffective or inappropriate. This project will advance the science of human–computer interaction and support the development of more accessible and privacy-preserving computing systems. The objective of this research is to design a robust, scalable unvoiced interface by leveraging the rising trend of sensor-equipped earables to track jaw motion and subtle facial vibrations for speech inference. This research will (1) develop computational models that map jaw kinematics and facial vibrations to phonetic and linguistic units; (2) design a personalized, context-aware closed-loop feedback control system that leverages structured knowledge schemas to adapt feedback in real time; and (3) build a low-power, earable hardware platform optimized for sensing jaw dynamics with support for programmable computational offloading. The resulting low-power, privacy-preserving platform will establish new foundations for silent human–computer interaction across diverse users and real-world environments. The research outcomes from this work will be integrated into a comprehensive education plan and will influence pedagogy at the intersection of cyber-physical systems, human–computer interaction, embedded sensing, and artificial intelligence. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.