CAREER: Synesthetic Teaching of Dexterous Robots through Multi-Sensory Fusion of Vision, Touch, and Vibration

Our world is messy, unpredictable, and teeming with diverse sensory stimuli. While meticulous modeling and ingenious engineering have enabled robots to revolutionize the factory floor, the ability to effectively manipulate everyday objects in common environments remains an elusive challenge. A fundamental reason for this challenge lies in their limited sensing capabilities. Consider the problem of unlocking a door lock. To accomplish this, humans utilize a range of senses including sound to locate the pocket with keys, vision to identify the correct key, tactile feedback to insert the key precisely, and proprioception to coordinate movements. These rich sensory modalities compensate for each other's deficiencies (e.g. visual occlusion), facilitate recall of past strategies (e.g. the click of unlocking) and ensure robustness against sensor failure (e.g. numb hands). Developing a robotic system that can synergistically utilize a wide range of sensors requires creating new algorithms capable of reasoning about high-dimensional, uncalibrated and noisy inputs. This Faculty Early Career Development (CAREER) project funds research that aims to develop Synesthetic Teaching, a new framework for teaching dexterous manipulation to multifingered robots using rich, multi-modal sensory observations. This research project is divided into four broad topics to advance the state of the art in robotics by integrating new sensing modalities for multi-fingered robots: (1) combining multi-camera streams with multi-fingered proprioception; (2) incorporating uncalibrated tactile feedback for enhanced offline imitation; (3) utilizing vibration feedback for adapting to dynamic manipulation; (4) developing multi-fingered robot systems capable of contact-rich manipulation in real-world scenarios created in the PI's lab. This research will create robots that solve more difficult manipulation problems than currently possible, by developing new algorithms in self-supervised representation learning, cross-modal fusion, and online imitation. The intended applications of these robots are in assistive robotics, flexible manufacturing, household robotics, fulfillment and others that require enhanced multi-fingered dexterity. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.