CAREER: Dexterous Multimodal Object Manipulation and the Curious Case of Tactile Shadows

This Faculty Early Career Development (CAREER) project supports research to realize intelligent and dexterous robots that seamlessly integrate vision and touch. In recent years, we’ve seen a dramatic rise in the role of data and learning in robotics. However, the vast majority of existing work focuses on vision and ignores the sense of touch. Touch provides robots with a wealth of knowledge from physical interactions and in many tasks it is crucial for success (e.g., reaching into a pocket or rummaging in through a grocery bag). Integrating touch with the rapid progress in vision will accelerate the deployment of skilled and reliable robots from research settings into the real-world. It will make fundamental contributions to transformative robotics technologies in practical applications, including manufacturing, healthcare, and home assistance – all tasks in which robots will need to interact with the world in some way. In this project, the researchers focus on tool-use as an application (e.g., in the kitchen for assistive care) and intend to develop methods that enable the robot to sense the tool it is grasping and adeptly manipulate it to perform a variety of skills. As part of the project, the education plan will integrate this research into undergraduate and graduate education through course development, the writing of a textbook, and research mentorship. The outreach activities will include an open-source software initiative and K-12 educational programs for high school students. The goal of this research project is to develop sensory-motor perception and control algorithms for dexterous object manipulation with visual and high-resolution tactile feedback. Central to this dexterity is the robot’s ability to reason over tactile signatures from sensors mounted at the mechanical interface between the robot and the grasped object (e.g., fingertips and palms). However, there are several fundamental challenges to overcome in high-resolution tactile sensing and touch-centric manipulation including how to interpret complex and high-dimensional tactile signals for state-estimation and how to leverage compliant sensor mechanics for contact-rich tasks. To address these challenges, this project will develop: i) a physics informed tactile representation that addresses "Tactile Shadows" -- an unstudied form of aliasing that affects contact patch estimation and can corrupt perception/state-estimation algorithms; ii) algorithmic tools for object pose and extrinsic contact estimation from active vision and touch that accounts for each modalities particular uncertainty; and iii) a control paradigm that exploits complex and compliant sensor mechanics for contact-rich and forceful tool-use. The union of these three aims intends to enable robust closed-loop object manipulation. These approaches will be evaluated in the context of three practical tasks for accuracy, robustness, sample-complexity, and task success. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.