RI: Small: Topology Guidance and Control of Neural Implicit Representations for Inverse Rendering of 3D Shape

The core computer vision problem of inverse rendering is concerned with recovering the shape and material of an object in spite of a lack of knowledge of camera position and lighting. A promising approach that has emerged in recent years uses neural-network based implicit representations of surfaces or volumes. However, it remains challenging to reconstruct shapes with complicated topologies such as 3d shapes with a variety of holes through them, especially when the resulting shape representations will be manipulated in detailed ways downstream. Nonetheless, being able to perceive and edit these kinds of complex topologies would enable significant new applications in 3D reconstruction, shape modeling, animation of non-rigid objects and editing complex 3D scenes. This project develops a collection of theoretical frameworks based on topological derivatives and neural homotopy to allow a level set evolution for surface reconstruction to satisfy user-defined properties on the topology, or allow for intuitive deformations. It develops a framework of topological derivatives in inverse rendering, beyond traditional shape derivatives, to allow the ability to introduce holes and other shape-changing modifications into implicit representations, which will mitigate the optimization difficulty for gradient descent approaches to reconstruct high genus shapes. It designs novel topology regularizations to induce deformations that are intuitive and amenable to user-defined constraints to enable reconstructing objects with non-rigid deformations. It also develops topology-aware estimation of the geometry of indoor scenes, while fully accounting for the material properties and complex light transport, which will allow handling layout changes like doors or window shades opening in applications like augmented reality. The project also develops a new computer vision and graphics curriculum at college and K-12 levels that incorporates experiential insights from technological applications of 3D reconstruction and inverse rendering. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.