CAREER: Digitize and Simulate the Large Physical World via Knowledge-Grounded Scene Representation

Numerous fields, such as agricultural robotics, necessitate that humans and intelligent agents have a shared understanding of the physical world to be able to make informed decisions. This requires that intelligent machines, to effectively assist humans, understand the physical world, simulate various possibilities, and predict and evaluate outcomes. Current computer vision systems, which largely focus on understanding and modeling observed phenomena, fall short of these capabilities. Similarly, domain-specific simulators fail to achieve this goal as they lack real-world grounding. This project aims to construct AI systems capable of creating a digital replica of the large 3D world and faithfully simulating various counterfactual scenarios, thereby enabling users to assess the outcomes of different decisions and actions. Central to this work is an actionable, knowledge-grounded scene representation, facilitating real-world modeling and simulation. The project has the potential to amplify advancements in various disciplines, such as robotics and agriculture. This project aims to achieve this goal via four directions that advance digitizing and simulating the physical world. To achieve this goal, the project offers a framework that will (i) create an actionable scene representation from real-world videos; (ii) infer physical parameters of the digital world from visual observations; (iii) perform physics-grounded simulation over the digital twin that enables the creation of realistic and accurate counterfactuals; and (iv) integrate the digital twin with domain knowledge and modeling for various applications. Based on the research studies, the project will develop a digital twin education platform that can be applied in undergraduate and graduate education, mentoring, and K-12 outreach activities to engage the next generation of researchers in computing. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.