CAREER: Towards a Theory of Deep Learning

Over the past decade, deep learning has evolved from conquering research benchmarks to systems that interact with humans on a daily basis, including in machine translation, healthcare, speech recognition, semi-autonomous vehicles, and automation. However, a large gap exists between its empirical successes and a theoretical understanding of why / when it works. This project aims to close this gap through foundational understanding of deep learning and designing algorithms to improve reliability and data efficiency. More broadly, the societal impact of this project include i) theoretical understanding and design of algorithms relevant to machine learning, ii) education plans that develop a new seminar series and workshops for secondary school teachers, and iii) improving disability accommodation in academia. This project is divided into three different thrusts. The first thrust is to understand the algorithmic regularization effect of algorithms and architectures. Using these insights, the team will design better loss functions and architectures to improve accuracy. The second thrust is to theoretically compare the accuracy of networks trained with stochastic gradient descent against their architecture-induced kernel methods. This comparison may theoretically demonstrate that neural networks can do feature learning, which explains the empirical success of deep learning, and that kernel methods cannot. Finally, the project will study representation learning, and theoretically analyze how deep networks can transfer their representations between different domains. Such a transfer will allow a reduction of the labeled data requirements for deep learning, potentially allowing its application to data-starved domains. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.