CAREER: Modeling Physical and Social Worlds for Embodied General Intelligence

Despite recent advances in embodied artificial intelligence (AI), achieving general-purpose intelligence (i.e., human-level intelligence) , where AI agents can flexibly perform like humans across a wide range of tasks across diverse environments, remains a fundamental challenge. Two persistent gaps hinder progress: (1) limited understanding of physical space, which constrains agents' ability to efficiently learn broad physical skills; and, (2) difficulty in open-ended social interactions, which limits multi-person (and agent) interaction scenarios. This project aims to bridge these gaps by developing embodied generalist agents capable of perceiving, reasoning, and interacting effectively with both the physical world and other agents in dynamic, evolving environments. In parallel, the project will support course development, student mentoring, and outreach activities, integrating foundational research with experiential learning to prepare the next generation of U.S. students for leadership in academia and industry. This project will focus on building robust physical and social world models for embodied agents by integrating physics engines with generative foundation models. The technical agenda includes: (1) developing paradigms that allow robotic agents to autonomously propose new tasks, generate corresponding environments, and acquire novel physical interaction skills through self-supervised simulation; (2) creating methods that combine open-world knowledge from foundation models with spatiotemporal memory and model-based planning, enabling agents to interpret goals, intentions, and social cues for effective collaboration in real-world social contexts; and (3) designing systems that co-optimize world model architectures and hardware to achieve real-time performance. Collectively, this project will establish a foundational platform for embodied AI and drive progress toward embodied general intelligence. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.