SHF: Small: An Integrated Architecture-System Framework for High-Quality and Cost-Efficient Learning-Based Super Resolution

Super resolution is a technology that enhances the quality of digital images by increasing their resolution, making them appear sharper and more detailed. This capability is important in a wide range of applications, such as medical imaging, satellite analysis, security monitoring, and immersive technologies like virtual reality (VR). However, current super resolution methods—many of which are based on deep learning—can be slow and require large amounts of computing power. This project aims to develop a new generation of learning-based super resolution that delivers high image quality, fast performance, and low power consumption. This project will have a transformative impact across a wide range of fields like medical imaging, entertainment, VR, surveillance, and autonomous vehicles. Moreover, by enabling sharper images with fast, energy-efficient processing, this research allows edge devices, such as drones and mobile phones, to perform real-time image enhancement without cloud reliance, saving bandwidth and power. This will drive more efficient processing in constrained environments, reduce costs for high-quality imaging, and unlock new possibilities for Artificial Intelligence (AI)-driven applications in smart cities and augmented reality. The project will expose more students to computing research through outreach, curriculum development activities, and disseminating research infrastructure for education and training. This project develops an integrated framework to enable high-performance, energy-efficient learning-based super resolution (SR). The research focuses on three key innovations: (1) extracting computer graphics (CG) information from GPU architecture during low-resolution rendering and integrating it into a convolutional neural network (CNN)-based SR as an intermediate feature, along with CG-guided architecture search and dynamic weight pruning, and architectural co-designs for high-fidelity, real-time SR processing; (2) improving Swin transformer-based SR on preserving fine details by utilizing patch entropy for dynamic window partitioning and replacing patch merging with splitting, supported by architecture-level optimizations; and (3) developing a multi-model framework that integrates proposed CG-guided CNN and details-enhanced transformer-based SR networks for excellent VR user experience on SR image quality and latency, with final evaluation on field-programmable gate arrays (FPGAs). This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.