CAREER: Sparse Sampling and Reconstruction for Rendering Through Per-Scene Optimization

Computer graphics rendering is increasingly becoming an integral part of our daily lives, in applications from product and architectural design to self-driving cars, to cite just a few examples. In the context of rendering, typically an image is generated through random sampling using what are known as Monte Carlo (MC) algorithms, simulating how the photons emitted from the light sources interact with the objects in the virtual environment and arrive at the camera. Accurate reconstruction of an image, however, requires simulating a large number of photons, making this approach costly in terms of computational power and energy consumption. This project will dramatically reduce the number of photons required to obtain a high-quality image, thereby significantly reducing the cost. This objective is achieved by novel ways of finding high-likelihood photon paths and by enhancing the quality of the image after rendering. Project outcomes will have broad impact on applications such as those mentioned above, as well as in areas beyond graphics that also rely on MC integration. Educational and outreach activities will use compelling real-world problems and concepts in computer graphics, particularly rendering, as motivational tools to get people of diverse ages and demographic backgrounds excited about STEM. This research will focus on accelerating the convergence of MC rendering by developing novel methods that operate during or after rendering. The two most common categories of methods used in this setting are importance sampling (during) and reconstruction/denoising (after). Unfortunately, the majority of existing importance sampling techniques provide path guidance locally and thereby ignore the model error on future points along the path. Moreover, current state-of-the-art reconstruction/denoising methods use a neural network trained on a set of noisy images and their corresponding ground truth, and the performance of these methods on test images that lie outside the distribution of the training data can be suboptimal. This project addresses these challenges by introducing novel frameworks that view these problems in a fundamentally different way. The specific research objectives are twofold: (1) To guide the paths according to distributions that are estimated in an error-aware manner, which requires introducing a global objective function that takes the entire path into consideration. (2) To design denoising systems that can be adapted to the test example at hand, by posing the denoising problem as an optimization problem and developing novel objectives that work effectively without the need for ground truth images. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.