CAREER: A Highly Effective, Usable, Performant, Scalable Data Reduction Framework for HPC Systems and Applications

This CAREER project researches and develops novel algorithms and software to improve the efficacy, usability, performance, and scalability of data reduction for high-performance computing (HPC) systems and applications. It contributes to the cyberinfrastructure (CI) of big data management for HPC applications in many domains such as cosmology, climatology, seismology, and machine learning. The research findings will be widely disseminated through open-source software packages and publications in premier conferences and journals. An integrated educational and outreach program is designed to foster CI workforce development, including integration of concepts and use of data reduction in curricula, research training for undergraduate and graduate students, and a specially designed training program for scientists and engineers from universities and national labs. This CAREER project simultaneously addresses these four critical issues in scientific data reduction through comprehensive analytical modeling and architectural performance optimization. Specific scientific contributions include: (1) it builds lightweight models to accurately estimate the compression ratio and quality of different techniques in the prediction and encoding stages of prediction-based compression, and optimizes the compression configurations to maximize the compression ratio under compression quality constraints; (2) it develops new efficient predictors and lossless encoding methods for lossy compression of scientific data on GPUs with deep architectural optimizations to achieve both high throughput and ratio; and (3) it deeply integrates the optimized compression with parallel I/O and MPI libraries with a series of optimizations to improve the performance of data movements and the scalability of HPC applications. The success of this research agenda enables scientists and engineers to well address the increasingly severe challenge of scientific data explosion. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.