CSR: Small: A Fine-Grained Hierarchical Memory Management System for Applications with Dynamic Memory Demand on GPUs

Graphics Processing Units (GPUs) have become an important component of modern high end computer platforms due to their ability to deliver high throughput, efficient computation. GPU utilization is therefore crucial to the performance of many applications. Despite over a decade of study in GPU programming, an important class of applications with dynamic memory demand is not well supported. This research aims to simplify the programming and improve the performance of dynamic-memory applications on GPUs. The outcome of this project will unlock the power of GPUs for a wide variety of applications with complex and dynamic memory usage, including bioinformatics, scientific computing, and machine learning. Additionally, this project will contribute to the development of high-performance computing courses and provide research opportunities to underrepresented students at the institution. There are mainly three challenges for supporting dynamic-memory applications on GPUs. First, the dynamic data sizes in these applications require data storage at multiple memory levels. Accessing and synchronizing data across the memory hierarchy can be complicated and time-consuming. Second, the dynamic memory consumption makes memory allocation nontrivial, as the amount of memory allocated to each data object at different memory levels can significantly affect performance. Third, many of these applications have complex and skewed memory access patterns, which makes it difficult to determine the optimal data placement in the memory hierarchy. This project aims to address the above challenges by introducing a fine-grained, hierarchical memory management system on GPU. The system provides a novel programming interface for managing multi-dimensional tensors in the GPU memory hierarchy, which will facilitate the development of dynamic-memory applications. Our system also provides automatic memory pre-allocation and memory access optimizations, which simplify performance optimization for these applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.