EAGER: IMPRESS-U: Random Matrix Theory and its Applications to Deep Learning

This IMPRESS-U project will be jointly supported by NSF, US National Academy of Sciences, and National Science Centre of Poland. The research will be conducted in collaborative partnership that unites the Pennsylvania State University, USA, Institute for Low Temperature Physics and Engineering of the National Academy of Sciences of Ukraine (ILTPE), and the University of Warsaw, Poland. The USA part of this IMPRESS-U project is co-funded by the Office of International Science and Engineering and MPS/DMS Applied Mathematics and Computational Mathematics programs. Part 1. The main ideas of Artificial Intelligence were formulated in 1970s, when complex technological processes in chemical, automobile and other industries became dependent on an enormous number of random and non-random factors. To answer this challenge, self-learning systems (SLS) were proposed. The SLS allow for fine tuning of production lines and adapting to constantly changing conditions (temperature, pressure, etc). One of the most promising modern realizations of SLS is Deep Neural Networks (DNNs) which are currently used everywhere from industry and defense systems up to internet technologies and cell phones. A key component of the DNNs’ learning process is adjusting the parameters of a DNN to increase its performance. These parameters form matrices with typically initial random entries. Hence, the application of Random Matrix Theory (RMT) could greatly benefit the learning process. In this project, it is expected to establish criteria for optimal learning via developing RMT tools. As a result, it will be possible to speed up the learning process, increase the DNN accuracy, reduce complexity, and avoid over-training. The project engages research teams from the US (Penn State University), Poland, and Ukraine. The project will build to a large extent upon the world-class strength in RMT of the school of mathematics in Kharkiv, Ukraine. The project will help to integrate graduate students, postdocs, and researchers from Kharkiv into the international scientific community and research workforce, and to prepare a new generation of Ukrainian researchers working in STEM fields. The Ukrainian students will be introduced to the state-of-the-art applications of RMT in DNNs. Part 2. Inspired by the function of biological neural networks, Deep Neural Networks (DNNs) have demonstrated their high effectiveness in a wide range of cutting-edge applications such as object, speech, and pattern recognition. However, they are still poorly understood from a theoretical point of view. Recent studies have shown that analysis based on Random Matrix Theory (RMT) can help to improve the convergence and learning speed of neural networks as well as improve accuracy and reduce the computational complexity of training algorithms in deep learning. The focus of the project is twofold: (i) further development of RMT techniques, (ii) employment of these techniques for developing numerical and analytical methods in deep learning. Specifically, the spectral properties of random matrix ensembles arising in the analysis of DNNs will be studied, both analytically and numerically. In particular, the project will address enhancing accuracy of DNNs and reducing computational complexity via RMT-based pruning and regularization techniques. To this end, the research will be concerned with finding an optimal weight initialization for untrained DNNs, determining stopping criteria for training, studying the nonlinearity’s effect on DNN learning speed, as well as justifying numerical and approximation methods and error estimates. The project brings together researchers from various fields and approaches in mathematics and physics. These fields range from deep learning, machine learning, RMT, and condensed matter physics to probability theory, and analysis. The project will develop analytical and numerical tools in RMT and deep learning and enrich the global mathematical community. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.