CRII: OAC: Deploy-First and Elastic Data Transfer Optimizations for High-Performance Networks

Scientific applications generate massive volumes of data through extensive simulations and physical experiments, requiring efficient transfer across globally distributed High-Performance-Computing clusters for further processing and collaborative research. These transfers support diverse scientific endeavors, including computational simulations and machine learning modeling. While high-speed networks manage this rapid data growth, current data transfer tools often fail to fully utilize these resources or focus excessively on network utilization using simplified architectures for quick convergence. These approaches frequently result in significant overhead on end systems and fairness issues. This project introduces a novel elastic design that maximizes unused resources without disrupting existing network and computing processes. It also adapts to the dynamic and evolving nature of computing clusters, where storage or network access patterns may shift unexpectedly. The project integrates advanced optimization algorithms that continually adapt to environmental variability, leveraging a comprehensive monitoring framework to capture real-time system dynamics and balance multiple objectives while tuning various transfer parameters. This project makes three key contributions: (i) it implements a robust monitoring framework to enhance system visibility, enabling global optimization and fair resource allocation; (ii) it uses the collected metrics to develop a simulator that replicates complex storage and networking dynamics, supporting offline and model-based training of generalized reinforcement learning agents; and (iii) it deploys pre-trained policies in production networks, continuously refining them to adapt to specific and evolving environments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.