FRR: Generative-Optimization Synergy for Policy Imitation in Contact-Rich Manipulation

Abstract Human hands manipulate objects with remarkable skill, constantly making and breaking contact to grasp, turn, and reposition items of many shapes. Robots still struggle with this ability, limiting their use in factories and homes. Artificial intelligence (AI) offers a promising path by enabling robots to learn manipulation skills from human examples. However, today’s methods often require people to operate robots for hundreds of hours to collect training data. The resulting skills can also be brittle: they may fail to adapt when tasks change, violate safety requirements, or offer users little control over how the robot behaves. This project addresses these problems by uniting two complementary approaches. Generative AI captures the rich variety of ways people handle objects. Model predictive control (MPC), a well-established method from control engineering, repeatedly replans robot motion in real time so the robot can adapt to changing conditions, satisfy safety constraints, and pursue task goals. By integrating these two methods, the project seeks to create a new control-guided AI paradigm for teaching dexterous robots from limited human input. Users can refine the robot's behavior through natural feedback, such as expressing a preference or physically guiding the robot. These capabilities could benefit custom manufacturing, space construction and maintenance, advanced prosthetics, and assistive robots in the home. They could also allow non-expert users to teach AI-enabled robots without programming expertise. The research is integrated with education and outreach, including new topics in the investigator’s online robotics materials, live robot demonstration events, and high-school robotics panels. These activities help students and the public better understand robotics and support preparation of a future robotics workforce. This project aims to establish a scalable, adaptive, and controllable paradigm for dexterity learning. The paradigm integrates the strengths of generative modeling and optimization-based control through three research thrusts. (1) A bi-level optimization framework is developed for scalable sim-to-real data generation from sparse demonstrations, such as a few images or key poses. The framework turns them into a large-scale contact-rich manipulation dataset with better transfer to real robots. (2) MPC-informed generative policy learning is introduced. It unites a generative model, which supplies broad manipulation strategies learned from human examples, with online MPC replanning, which ensures safety and adaptive planning in dynamic environments. (3) A controllable and robust policy fine-tuning framework with human feedback is designed. Users give runtime feedback, such as preferences and physical corrections, to adapt the policy for different task goals and safety constraints. The framework is designed to remain effective even when human feedback is noisy or inconsistent. All methods are evaluated in simulation and on robot hardware. Tasks include reorienting objects held in a multi-fingered hand, non-prehensile manipulation, and functional grasping with help from the surrounding environment. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.