CAREER: Acoustic Ambient Computing: Algorithms, Architectures, and Prototypes

Acoustics play a crucial role in connecting humans to computing infrastructure. Voice assistants have created a natural means of interacting with IoT devices; smart earphones are emerging with sensing and assistive skills; and acoustic augmented reality is around the corner. This proposal envisions a new frontier in this computing space by enabling advanced acoustic sensing and inference on low-power, battery-free computing platforms embedded in everyday environments — creating a paradigm called acoustic ambient computing. The paradigm aims to comprehend human activities, motions, gestures, and behavior through ambient, infrastructure-embedded sensing. Target applications include data-driven health technologies and physical recovery tracking in hospitals, immersive assistance at workshops, positive behavioral reinforcement in offices and collaborative workplaces, and even integration in open spaces such as parks or bus stops. This research focuses on developing theoretical building blocks, algorithms, and hardware-software prototypes to realize low-cost, scalable, and sustainable solutions for sensing human activities and interactions using sound as a ubiquitous sensing modality. The project efforts are organized as three research thrusts: (a) The first thrust enables sensing of large-scale gestures and movements near everyday surfaces, like walls, floors, and tabletops. It is realized through a novel coded signal projection method using passive micro-structures at the physical layer and an adaptive intermittent computing architecture. (b) The second thrust zooms into micro-meter level motion and micro gesture detection. This research module explores novel cross-spectrum sensing and localization techniques and a lightweight neural attention-based selective resolution approach for processing. (c) The final research thrust introduces the notion of physical context to voice and speech interactions across indoor and outdoor movements, gestures, and micro-activities. This project will establish the critical components of a wearable-free paradigm to interweave sensing and inference within everyday environments so that computing services can be effortlessly blended into people’s daily lives. The developed platforms and prototypes will foster a range of new applications in interactive home automation, assistance and healthcare, collaborative workspaces, and home security. It will help create frameworks for retrofitting existing buildings and augmenting new indoor and outdoor environments to turn them into interactive smart spaces. The proposed research examines the interplay of embedded intelligence, wireless sensing, low-power networking systems, and human-computer interaction. Insights from this project will also lead to new learning modules used to teach the fundamentals of device-free computing to students. Moreover, this project will introduce the concepts of low-power embedded intelligence to undergraduate and K-12 students. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.