Collaborative Research: Hardware-Aware Matrix Computations for Deep Learning Applications

Deep Learning (DL) systems these days are ubiquitous, and arguably affect our everyday lives more than any other computational system. Recently, such deep models (e.g., GPT-3) have increasingly become large and unwieldy with a large computational footprint. Given the ever increasing computational requirements, it has become nearly impossible to make progress on cutting edge research in learning such DL models outside of a few large technological companies. This project will explore principled ways to create DL systems that are as expressive as the large deep models but at a fraction of the computational cost. On the practical front these improvements are expected to expand the possibility of creating such powerful DL models to larger parts of society. On the educational front, this project will train undergraduate (UG) researchers and will integrate responsible computing into UG curriculum. This project will study how one can use structured matrices in concert with modern hardware constraints to achieve similar performance as these really large models but at a fraction of size and computational cost. Specifically the investigators focus on the following two thrusts: (i) Design the ‘holy grail’ of structured matrices that satisfy all properties that are desirable in DL applications (including having an efficient projection problem as well as having efficient parallel and/or hardware friendly learning algorithms); and (ii) Thinking of new applications that our new theory can unlock. This DL lens exposes new problems to consider when studying structured matrices. In turn, the new family of structured matrices studied in this project will not only have immediate practical applications but will also unlock new twists on classical theoretical problems in matrix computations. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.