ERI: Exploring Visual-Haptic Synergy in Fragile Object Manipulation

This Engineering Research Initiation (ERI) grant will support research that contributes new knowledge to robotic manipulation, addresses the current bottlenecks in manipulation reliability, improves work efficiency, enhances people's well-being, and consolidates the United States' leading position in robotics. Medical, agricultural, and household robots face a common challenge: the reliable and safe manipulation of fragile objects. Overcoming this bottleneck will enable significant advancements in the application of robotics in real-world environments. However, the variety of object properties and differences in manipulation tasks in the real world hinder the application of manipulation methods that are fully validated in laboratory environments. This award supports fundamental research to provide new knowledge for reliable object manipulation. This new knowledge will lead to the development of a technology that can automatically adapt to a variety of objects and manipulation goals by understanding the relationships between object properties, manipulation tasks, and robot operations. This technology will achieve breakthroughs in the safety, reliability, and efficiency of object manipulation in robotic surgery, agricultural robots, and household robots, directly benefiting the United States' medical, economic, and social fields. This research will also provide learning and research opportunities for students with disabilities, promoting balanced societal development and comprehensive engineering education. Integrating vision and touch is considered the solution to the problem of manipulating deformable and fragile objects. Existing solutions directly integrate visual and tactile information to guide manipulation. Although simple in theory, these solutions tightly couple perception, modeling, and control issues, making it difficult to overcome the resolution differences of multimodal perception, solve the problem of sensor synchronization, and achieve universal applicability. This research project will work to establish an understanding of the complementary relationship between touch and vision and use novel deep causal learning techniques to decouple multimodal perception and the modeling of object properties. This approach will strive to realize a control method based on object properties that can simultaneously optimize the position, angle, speed, acceleration, and interaction force, thereby achieving reliable manipulation of deformable and fragile objects with wide applicability. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.