Elements: Adaptive End-to-End Parallelism for Distributed Science Workflows

Technological advancements in sensing and computing technologies have led to an unprecedented increase in the amount of data generated by scientific applications. As science projects are increasingly distributed in nature, the increase in data sizes in turn results in an increased volume of traffic that needs to be moved across geographically distributed locations. Although significant investments have been made to build high-speed networks to facilitate data movements between research and education institutions, it is difficult for domain scientists to efficiently utilize this available capacity mainly due to the lack of scalable data transfer services. This project addresses this need by developing a scalable and reliable data transfer service. It further integrates the data transfer service into elastic workflow management systems to achieve end-to-end optimization for distributed science workflows. This project makes three novel contributions to the field: (i) it innovates scalable integrity verification and encryption for file transfers to ensure the reliability of file transfers without sacrificing performance. It takes advantage of computing resources available at data transfer nodes to scale the performance of integrity verification and channel encryption features. (ii) It innovates end-to-end parallelism for distributed workflows by integrating an online transfer optimization service into elastic workflow management tools. Unlike existing workflow management solutions, which merely focus on the optimization of computing tasks, the proposed integration of online transfer optimization services into elastic workflow schedulers enables true end-to-end parallelism for distributed workflows. (iii) Finally, it demonstrates the performance of the developed service on a real-world bioscience workflow that streams a large volume of sequence read archive data from the NCBI database to extract computation-ready SAM/BAM files. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.