Scalable Robot Validation and Data Creation with Compositional Generative Simulation

Robots that are capable of performing complex tasks in everyday environments and safely interacting with humans have the potential to greatly benefit society. The benefits include such things as providing assistive care to older adults to collaborating with in industrial settings. However, developing such capable robots remains extremely challenging, as it requires testing and improving robot systems across an enormous variety of possible real-world situations. This research project aims to transform the robot development process by creating a powerful simulation framework that can automatically produce diverse, realistic virtual scenarios for training and evaluating robots. The researchers will develop novel machine learning techniques to simulate various ways that humans would interact with robots, create diverse and realistic 3D environments, and generate artificial sensory data that closely mimics what robots would experience in the real world. The research will enable robot developers to rapidly test and refine their systems across millions of virtual scenarios before physical deployment, potentially accelerating the development of safe, capable, and versatile interactive robots while significantly reducing development costs and risks associated with real-world testing. The research addresses the limitations of current scenario-based development in robotics, which is costly, time-consuming, and difficult to scale due to the need for physical deployments in diverse environments. To overcome these challenges, this project aims to shift the bulk of robot validation and data generation to simulation. Specifically, the research will develop a novel compositional generative simulation framework that integrates three key components: (1) a generative model for long-horizon interactive behaviors of non-robot participants; (2) a model for generating dynamically- and geometrically- consistent image and multimodal sensor observations; and, (3) a scene graph generator for static and dynamic objects and their layouts. The integrated simulator will produce a broad range of scenarios, including long-tail and safety-critical events, with user-specified levels of granularity. The team will evaluate the framework by using it to validate existing robotic systems and to generate training data for a learning-based mobile manipulation system operating in realistic home environments. This approach aims to significantly improve the scalability and effectiveness of robot development processes, potentially accelerating the deployment of robust and versatile robotic systems across various domains. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.