Collaborative Research: OAC: Core: Cross-layered Data Management to Reduce Data Movement in HPC Workflows

This project addresses the growing challenge of data movement in high-performance computing systems with diverse processors, memory, storage, and networks. These systems are critical for national-scale efforts in drug discovery, materials science, energy research, and large-scale artificial intelligence and machine learning. Applications such as molecular dynamics, graph neural networks, and particle-in-cell simulations generate large data volumes that must be moved efficiently. Data transfers often limit performance rather than computation. This project develops tools to reduce data movement time and energy, improving throughput, efficiency, and scientific productivity. It empowers researchers and developers to scale workflows on complex HPC systems while fostering collaboration among academia, industry, and national laboratories to transition ideas into practical solutions for exascale platforms. The project also advances national interests by enabling scalable AI and simulation workflows and engaging students in systems research for next-generation infrastructure. The project develops a unified framework to reduce data movement overheads in heterogeneous high-performance computing systems. It integrates three core components: a cross-layer monitoring and learning framework that characterizes data transfer patterns and predicts contention; a heterogeneity-aware data movement scheduler that coordinates bandwidth usage across computation, memory, storage, and interconnect resources; and a collaborative caching and prefetching architecture that anticipates future data needs across workflows. The framework treats data movement as a first-class task, parallel to computation, and uses analytical and machine learning techniques to reduce interference and improve overall throughput. The research is validated through representative workloads on petascale and exascale systems, including simulations and machine learning pipelines. Results will provide generalizable strategies for optimizing data movement in next-generation scientific computing environments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.