SHF: Small: Synthesizing Mixed Discrete/Continuous Programs with the Neurosymbolic Librarian

This project concerns automatically generating computer programs that can use neural networks and other machine learning models as subroutines. Programs like these are important because they form the cornerstone of modern machine learning systems, and also because symbolic programs and neural networks are complementary in their abilities, so such systems could learn to solve many diverse problems using a mixture of neural networks and symbolic code. However such programs are difficult to automatically generate or synthesize. The project’s novelties are new strategies that makes it much easier to generate such programs by using machine learning. The project's impact is a step toward systems that could learn to solve new problems using a mixture of neural networks and symbolic code, as well as a step toward Artificial Intelligence (AI) systems that could assist the development of further AI systems. From a technical perspective the project presents a way of jointly generating symbolic code and neural network weights both using gradient descent, by relaxing the discrete space of symbolic code into a continuous form. Because convergence of this relaxation can be difficult, the investigator proposes learning to generate the code in a multitask setting, which allows learning across many problems to aid convergence. The results will be showcased on generating 3-dimensional graphics programs, mixing implicit neural representations of geometry with discrete graphics primitives, as well as a few-shot learning domain. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.