Collaborative Research: Elements: Computational Storage Virtualization for Accelerating Data-Driven Scientific Applications on Supercomputers

With the emergence of computational storage devices and smart storage solutions, new system-level support is needed to enable data-driven scientific applications to efficiently access and utilize underlying compute and storage resources. This project contributes by creating a new basic toolchain and system technologies for such systems and by evaluating them with representative scientific applications and data analytics. It has the potential to impact other scientific domains because data-driven scientific applications are used in many scientific and engineering domains, including national security, power system reliability, and food security. The project will provide research training in Very Large Scale Integration (VLSI) and AI for undergraduate students and provide them with an educational pathway to pursue advanced degrees. This collaborative and interdisciplinary project seeks to bring experts in computer systems, field-programmable gate arrays (FPGAs), high-performance computing (HPC), and domain scientists together to design and implement a virtual computational storage system λ-HDF5. The project is driven by real use cases and built on state-of-the-art HPC and machine-learning cyberinfrastructure. The project has three specific goals. First, the project will develop an HDF5-compatible interface and provide support for a wide variety of computer kernels. Second, it will identify, dispatch, and execute compute kernels on faster devices across multiple I/O layers in HPC systems. Third, it will efficiently manage co-located raw data and pre-processed data on distributed and heterogeneous storage devices. Scientific applications using λ-HDF5 for data management can benefit from accelerated data ingestion pipelines. λ-HDF5 also makes computational storage devices more accessible to users in scientific computing communities. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.