CISE-ANR: HCC: Small: Omnidirectional BatVision: Learning How to Navigate from Cell Phone Audios

This project aims to develop real-time 3D space reconstruction from sound captured not by expensive specialized equipment, but by common-place consumer-grade mobile phones. The approach, inspired by echolocation used by bats, is to develop from sound alone 3D spatial maps that are sufficient for navigation, such as close obstacle avoidance and finding distant exits in a crowded train station. The research will enable 3D vision beyond the line of sight and in low or no light conditions with applications ranging from listening cars that can hear pedestrians around the corner to collective 3D map reconstruction from crowds. Project outcomes will contribute to better computational modeling of sound perception and effective sound-vision integration in robotics, as well as to impactful applications such as navigational aids for visually impaired persons and for fire-fighters in low visibility conditions caused by smoke or darkness. The work will provide a complementary cost-effective alternative to visual 3D mapping that allows everybody to become a 3D content creator. The task of 3D perception from sound is challenging. While stereo audio provides direct cues for horizontal direction of arrival estimation, it only works in well controlled environments. There are no simple mathematical models to map sound to 3D space in real-word settings, as many factors such as device orientations, room layouts, materials, background noises shape sound propagation. This project takes a machine learning approach to infer 3D space from cell phone audios. A large-scale audio-visual dataset will be collected in different environments using a sensor-rig with a binaural microphone, a speaker and an RGB-D stereo. An attached smartphone will record time-synchronized data with its own stereo microphone and cameras. The speaker will emit signals to enable echolocation, but a part of the data will contain only naturally occurring sounds. Several indoor and outdoor environments with LiDAR scanned 3D models will serve as ground-truth. Data will also be collected in public streets to test robustness in realistic situations where LiDAR scans are not possible. Given the dataset, several 3D scene reconstruction tasks will be formulated for both the field of view and full 360° view, first with privileged sensor data and finally from cellphone sensors alone. After collecting large-scale audio-visual data in a variety of environments with binaural microphones and stereo cameras, a model will be trained to map sound data to depth maps extracted from visual data. Once the model is trained, it will be able to “see” the 3D space based on sound inputs alone. The model will then be adapted to achieve the same high quality 3D perception with stereo-microphones and sensors available on a mobile phone. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.