DMS-EPSRC: Asymptotic Analysis of Online Training Algorithms in Machine Learning: Recurrent, Graphical, and Deep Neural Networks

Neural network models in machine learning have achieved immense practical success over the past decade, revolutionizing fields such as image, text, and speech recognition, engineering, medicine, and finance. The training algorithms used for these complex machine learning problems are successful in practice, but they are often ad hoc. Mathematical theory is yet to be established in many cases and there is the potential to improve training algorithms and models via rigorous analysis. The primary purpose of this research is to develop a rigorous mathematical analysis for the training algorithms for neural network models used in several key areas of machine learning. Developing and testing mathematical theory for widely-used training algorithms is crucial for ensuring their reliability and guaranteeing their performance in practice. This research project is integrated with an educational program that is designed to help in the training of undergraduate and graduate students in applied mathematics, engineering, computer science and data science in the exploration of training algorithms, their analysis and their performance. This project involves the development of new mathematical approaches for a rigorous analysis for the training algorithms for feedforward, recurrent and graph neural networks by rigorously deriving McKean-Vlasov type of mean field limits, central limit theorems, statistical scaling limits and large deviation principles. This will be achieved by leveraging methods from stochastic analysis and weak convergence theory to study the asymptotics of online, stochastic training algorithms and neural network models as the number of hidden units becomes large. The project also involves making direct connections between the mathematical analyses and parameter initialization, hyperparameter selection, the design of optimization/training algorithms, and model selection. In addition to proving convergence theory for important neural network training algorithms, the research will be of interest outside of machine learning as it will study a new set of mean-field problems with novel and mathematically challenging features, making the methodology of interest to other fields including mathematical biology, physics and engineering. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.