Physics-Composed Generative Policies: Embedding Vision-Native Gaussian Contact Physics for Reliable and Transferable Autonomy

Robots are increasingly taught new skills through generative policies: artificial intelligence (AI) models that learn from task examples and then generate robot motions, much as modern generative AI produces images or text. While this approach has produced impressive demonstrations, it has a critical blind spot: generated motions may look plausible while failing to respect the physics of touch. A robot may jam during insertion, slip at the wrong moment, or press too hard, because a millimeter of error or a small excess of force can separate success from failure. Today this gap is often addressed with large amounts of high-quality training data and repeated trials, making capable robots expensive to develop and unreliable to deploy. This project takes a different path: it builds the laws of physical contact directly into the architecture of generative policies, so that generated motions are physically valid by construction. The result is robot learning that requires less data, produces actions that are more physically reliable, and adapts when robots, objects, or contact dynamics change. These advances lower the cost and improve the trustworthiness of robot autonomy in contact-rich domains such as flexible manufacturing, warehouse automation, and surgical assistance. The project's education and outreach activities include an open-source interactive robotics textbook, hands-on robotics workshops for K-12 students, and industry-connected mentoring, preparing students and the broader public to responsibly deploy robots that act in the physical world. The research addresses a structural limitation of generative policies, such as diffusion-based policies: a generative policy itself has no notion of physical validity. The goal of the project is to establish physics-composed generative policies, a new class of generative models for contact-rich robot autonomy in which contact mechanics operates as a native layer of the network architecture, rather than as an external simulator, training environment, or corrective filter. The research has three thrusts. First, a contact mechanics model is developed to be embedded into a generative policy network: the model operates on vision-compatible scene representations, enabling end-to-end differentiation from camera pixels, through contact interactions, to generated actions. Second, a learning framework for physics-composed generative policies is developed, in which action generation and contact feasibility are jointly coordinated to balance motion diversity with physical consistency. Third, task-level contact representations are developed to capture how useful interactions unfold over time, supporting skill adaptation when robots, objects, or contact dynamics change. The methods are evaluated on real-world manipulation tasks, including precision assembly, tool use, and object reorientation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.