Sound Navigation: Enabling Tiny Robots to Find Their Way Through Smoke, Dust, and Darkness

This award will support fundamental research to enable aerial robots smaller than 100 millimeters and weighing less than 100 grams to navigate through cluttered surroundings in the presence of smoke, darkness, dust, fog, and snow. To accomplish this in a completely self-contained way, the robots in this project will use sound waves instead of light to sense nearby objects. Sound waves penetrate much farther than light through airborne particles, but are easily confused by propellor noise and are unable to reliably distinguish small features. Through advances in mathematical modeling, neural network design, and sensor characterization, this project will greatly improve the quality of sound-based images, without exceeding size, weight, and power constraints imposed by the limitations of the small aerial platform. The research will result in inexpensive and easily transported drones with the potential to save lives in forest fires, cave rescue, wildlife management, and natural disasters, thereby contributing to the health, well-being, and economic strength of the Nation. The research brings together multiple fields including robot perception, bio-inspired artificial intelligence, sensor fusion, and signal processing, thus germinating new research questions and creating research training opportunities across traditional disciplinary silos. Aerial robot navigation built on multi-modal visual-sonic-inertial (VSI) sensing can overcome challenges in visually degraded and challenging scenes that may confound current sensing systems such as may arise due to darkness, snow, smoke, dust, or fog. This project addresses scientific barriers to widespread deployment of VSI-based systems, through the advancement of representations, mathematical models, and neural architectures. These advances will allow effective extraction and fusion of information from VSI systems, without reliance on any external infrastructure or prior information about the environment, while respecting practical constraints on on-board computation and information processing. The project builds on the following three key ideas: (1) enhancing ultrasonic depth estimation through bio-inspired passive structures, active noise reduction techniques, and physics-informed neural networks that leverage robot motion to overcome the ill-posed nature of sparse sensor trilateration; (2) fusing multi-modal sensor data using uncertainty-aware deep learning models and factor graph optimization to achieve robust depth estimation across varying environmental conditions; and (3) implementing a hierarchical reinforcement learning navigation stack with compositional policies for perception-action synergy, obstacle avoidance, and goal-directed movement. The approach combines mathematical modeling of wave physics with deep declarative networks, employs sim2real2sim methodologies for robust generalization, and utilizes factor graph formulations for temporal consistency. The results will be extensively tested in controlled physical experiments simulating autonomous operations in (1) dense forest with low light and heavy fog, (2) forest fires with corresponding thick smoke, and (3) caves, mines, and collapsed structures subject to high dust and low light. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.