Collaborative Research: HCC: A General Neural Light Transport and Material Framework

Today, realistic image synthesis or rendering software creates computer graphics images of real and imaginary worlds for movies and games, helps design buildings and plan renovations, and lets engineers see the products they are creating. To create images that truly depict the world as it would appear in reality, the users of rendering systems have to build elaborate models that include every detail that could contribute to visual appearance. In a human character every hair fiber is represented explicitly; in a meadow every blade of grass is its own separate object. Creating these models is slow and expensive, and rendering them requires a substantial amount of computation time. This project explores a fundamentally new way to represent scenes using machine learning, which learns a description of how objects reflect light directly from real and simulated measurements. These new models will be able to create visually realistic worlds with less complexity in the model, encoding how objects look without having to model very small structures explicitly. The learned model also will provide a single representation for all kinds of objects and materials. This representation is the basis for two further aspects of the project. One goal is to directly predict how light reflects through the scene, without explicit simulation. A second goal is to create generative AI tools that start with coarse models, text, and images, and generate fine-scale scene detail automatically. Together this project aims to transform the way materials and light transport are represented in computer graphics, enabling downstream applications including industrial design, entertainment, advertising, home remodeling, and architecture. The project will also support overhauling widely used classroom content to reflect the impact of machine learning on computer graphics and to introduce these topics through research projects for early-stage undergraduate researchers. The path to develop this new approach involves replacing, one by one, expensive physical models for light transport and materials with compact neural abstractions. Starting with the transport or propagation of light in a scene, neural approaches for light transport will be developed. The project will use data computed by path tracing (a method to explicitly simulate light transport) to train a machine learning transformer on point clouds, which can be applied for general scenes and predicts the light distribution due to global illumination. On the material side, neural materials will be extended from conventional opaque objects to subsurface scattering, where light scatters within the material, and ultimately to a voxel-based light field scattering model that can approximate any kind of scene content. This will lead to a unified neural representation for geometry and materials in which objects and scenes are not represented explicitly but rather abstracted into a model for how they scatter light locally. These unified models in turn will define a latent space for local scattering behaviors that can provide a richer form of input for improved transformer-based global illumination models. Neural light transport and materials also offer new opportunities in content creation, including novel generative appearance models that can take in a simple text prompt or image to create rich visual appearance of natural or synthetic worlds, and can fill in detail conditioned on existing larger-scale content. The ultimate goal is to easily specify and manipulate appearance from the smallest visible details all the way up to full virtual environments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.