CRII: OAC: Intelligent Error Control for Multi-tier Progressive Data Management

Advances in computing capability have led to huge improvements in the resolution and validity of scientific simulations, which has enabled science breakthroughs that were not possible in the past. With the sheer amount of data generated, analysis has become increasingly cumbersome due to the high cost of data storage, transmission, and computing. Error-controlled progressive data management frameworks have been designed to address such issues but have a number of limitations. To overcome these limitations, this project develops intelligent error control for the progressive data management framework on multi-tier storage systems. The effort advances next-generation data reduction and management techniques for advanced cyberinfrastructures, benefiting scientific users and expediting discoveries. This project develops an efficient progressive data management framework that controls reconstruction errors in fine-grained areas. The framework optimizes the data retrieval mechanism based on the interplay between error control factors and I/O size. Such optimization is designed as a derivation-free heuristic searching algorithm to identify the satisfactory error control factors towards the minimal I/O. The framework supports adaptive tuning of error control factors through a surrogate-based model that can identify the best configurations tailored for every application under the constraints of error tolerance and quantities of interest preservation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.