HCC: Small: Transformation-driven Material Appearance Modeling

Realistic digital 3D representations of imaginary worlds, as well as the world around us, support many downstream applications ranging from civilian and military virtual training simulators (for example, flight simulators to train pilots and virtual fire drills for first-aid responders) to applications that help to navigate and explore unfamiliar places (for example, Google Street View). Modeling the appearance of realistic, rich, complex materials is an essential component OF creating realistic virtual worlds; without realistic looking materials, the virtual world will feel synthetic, which in turn reduces the immersive effect of applications. Creating digital representations of real-world materials remains a labor-intensive task that often requires skilled artists and specialized measurements. This project aims to democratize this process by creating a new AI-enhanced framework for authoring rich digital materials from ordinary photographs of the target material. Furthermore, this project supports workforce development and transferability of skills through training of graduate students in AI, computer graphics, and computer vision. The central hypothesis of this research agenda is that digital material creation can be made substantially more efficient by starting from a reasonable approximation and then progressively refining it towards the desired material appearance. Inspired by recent learned image-to-image (Img2Img) transformations in image processing, this project will investigate learned material-to-material (Mat2Mat) transformations that convert one digital material representation into another. The main research thrust aims to explore different types of transformations and leverage generative models to frame Mat2Mat transformations as sampling from a conditional distribution. The overarching vision is a transformation-driven material appearance modeling framework where an input image is first transformed by an Img2Mat transformation and then modified to meet the users' expectations via a sequence of Mat2Mat transformations. In addition to the main research thrust, this project also endeavors to address shortcomings in current diffusion-based Img2Mat models by better leveraging the embedded priors in pretrained generative image diffusion models, as well as investigating automatic transformation inference and appearance models beyond spatially-varying reflectance distribution functions. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.