I-Corps: Active Acoustic Sensing for Wearables

The broader impact/commercial potential of this I-Corps project is the development of active acoustic sensing technology. Currently, there are over a billion wearable devices in use and rising, including smartwatches, smart glasses, and headphones. However, they lack the capability to continuously monitor human body pose and facial expressions. The proposed technology enables wearable devices to track human body pose and facial expressions, addressing a gap in the market. In addition, this technology facilitates activity recognition through wearable devices, aiding in tracking things such as eating habits and diet monitoring. The proposed technology may be used for medical applications, including the restoration of speech capabilities for individuals with vocal cord damage, providing an interface for computer interactions for people with disabilities, and remote health monitoring of patients. It also may enhance user experience by creating a more human-centered interface for wearable devices, simplifying computer device control, and may broaden the range of applications for wearables such as smart glasses. This I-Corps project is based on the development of a low-power, privacy-secure ultrasound sensing technology that integrates with wearable devices. The proposed technology enables wearable devices to monitor human body pose, facial expressions, and eye movements. The technology utilizes two pairs of miniature microphones (approximately 5 mm) and speakers (approximately 6 mm) that are commonly found in wearables, optimizing space utilization. The technology employs inaudible ultrasound waves (frequency > 20KHz) emitted by the speakers that are reflected/diffracted by the human body, and the microphones capture relevant data. Data collected by these wearable devices may be used for various applications. Machine learning algorithms then process this information to predict body pose and facial expressions. Importantly, since only ultrasound waves are used, there is no processing of spoken words from the device wearer, ensuring no disruption to the user's daily activities. In addition, the technology is energy efficient using just 143 mW and providing over 20 hours of operation on a Google Glass battery. This sensing technology has been integrated successfully into various wearable device form-factors, including rings, wristbands, glasses, earphones, and headphones. Multiple functional prototypes have been developed and tested, and research results have been published that demonstrate the technologies utility. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.