Collaborative Research: RI: Medium: Principled Approaches to Deep Learning for Low-dimensional Structures

The resurgence of deep neural networks has led to revolutionary success across almost all areas of engineering and science. Despite recent endeavors, current theoretical understandings of deep networks remain fragmented and only pertain to idealized and over-simplified network models. There is a significant lack of a systemic and unified approach for designing and explaining deep networks. Therefore, the underlying principles behind the success of deep learning still largely remain a mystery, which hinders its further development and adoption to broader applications. Nevertheless, the blessings of dimensionality imply that real-world data often reside in low-dimensional structures, and ample empirical evidence implies that there is a strong connection between deep learning and low-dimensional modeling. This connection implicitly appears in many different forms, in terms of learned representations, network architectures, and optimization strategies. However, these connections are far from being elucidated nor are they fully exploited. Based on the theory of data compression and optimal coding for learning from low-dimensional structures, this project aims to bridge the gap between the theory and practice of deep learning by developing a principled and unified mathematical framework. To develop this framework requires two steps. First, this project will design white-box deep networks by unrolled optimization schemes for maximizing the information gain of the resulting representation, which can be measured precisely by the coding rates of the representation. Second, the project will guarantee correctness through rigorous mathematical analysis of the optimization objective for learned representations. Third, this project will ensure consistency of the learned representations through a self-correcting closed-loop transcription framework that integrates encoding and decoding into a complete autonomous learning system. This new framework naturally unifies representation learning for all purposes: discriminative, generative, and auto-encoding, and is generalizable to all settings: supervised, unsupervised, self-supervised, and continuous learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.