CRII: OAC: Enabling Quantities-of-Interest Error Control for Trust-Driven Lossy Compression

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Scientific simulations and instruments are producing data at volumes and velocities that overwhelm network and storage systems. Although error-controlled lossy compressors have been employed to mitigate these data issues, many scientists still feel reluctant to adopt them because these compressors provide no guarantee on the accuracy of downstream analysis results derived from raw data. This project aims to fill this gap by developing a trust-driven lossy data compression infrastructure capable of strictly controlling the errors in downstream analysis theoretically and practically to facilitate the use of data reduction in scientific applications. Success of this project will promote the progress of science in multiple disciplines via effective data reduction, and contribute to resolving important societal problems including electric generation, weather forecasting, material design, and transportation. Moreover, this project will contribute to the growth and development of future generations of scientists and engineers through educational and engagement activities, including development of new curriculum and recruitment of K-12 students. Existing lossy compression techniques either overlook error quantification or provide error control only for raw data, leaving uncertainties in the outcome of downstream quantities of interest (QoIs) computed from the raw data. This greatly concerns many computational scientists who wish to reduce their data while preserving necessary information, preventing them from adopting lossy compression in their applications. This research will address these problems through an integration of theory and implementation via three tasks. First, a novel theory enabling error control on downstream QoIs will be developed. This will fundamentally address the trustability issues of existing error controlled lossy compressors that provide error control only on raw data. Second, an optimization method ensuring tight error control will be applied based on rigorous analysis, to achieve higher compression ratios under the same requirements. Third, a scalable infrastructure will be built through a careful integration with advanced compression frameworks and tailored parallelization based on target QoIs, in order to take full advantage of existing compression algorithms and computational patterns in the target QoIs. The project will enable application scientists to store the most valuable information in their data based on their unique needs, creating opportunities for novel findings in multiple scientific disciplines including climatology, cosmology, and seismology. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.