CAREER: Physics-Based Differentiable and Inverse Rendering

This project will develop new computational tools to infer physical parameters such as object shape and optical properties of a scene from measured images such as photographs. These tools are essential for building digital twins of real-world objects and will enable new applications in a wide array of fields including computer vision, computational imaging, robotics, and virtual/augmented reality (VR/AR). Unlike many existing methods that are purely data-driven, this research will develop inference techniques that leverage a simulator of how light propagates. This simulator will be differentiable, meaning that it is possible to smoothly relate its control parameters to its decisions, offering a new level of generality and physical accuracy for recovering parameters reliably under complex scenarios such as illumination from reflected light. Project outcomes have the potential for broad impact by creating new areas in computer graphics and computational photonics. Additional broad impact will derive from the PI's commitment to promoting STEM education for underrepresented minorities, and from the project facilitating UCI’s outreach programs at the undergraduate and high school levels including lab visits to allow hands-on experience with software development. This research will enable differentiable and inverse rendering that is efficient, physically accurate, and sufficiently general to handle arbitrary scene parameters under a wide variety of light transport phenomena. The work will make the following four core contributions: first, devising new mathematical tools to describe how infinitesimal changes in a virtual scene affect the distribution of light, supporting a variety of light transport models including steady state, polarized, and transient; second, introducing physics-based differentiable rendering algorithms that enjoy the generality of the new formulations while providing low-variance derivative estimates; third, leveraging these algorithms to build differentiable rendering software systems capable of efficiently handling complex real-world configurations; and, lastly, utilizing the new algorithms and systems to introduce physics-based inverse rendering pipelines that offer a new level of generality and accuracy benefiting a wide array of applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.