SHF:Small:Intelligent Management of Hybrid Workloads for Extreme Scale Computing

The high-performance computing (HPC) community is embracing artificial intelligence (AI) techniques for countless pursuits, from driving ground-breaking scientific discoveries to protecting our national security. As newly emerging machine learning and date-centric workloads proliferate in HPC, current workload-management systems cannot keep up with the significant challenges introduced by the diverse mix of applications co-running on heterogeneous systems. This project tackles the problem by developing new workload-management methods to catalyze the convergence of HPC, AI, and data analytics. It will develop fundamental improvements in HPC workload management to promote the use of large-scale supercomputers for emerging data-centric applications (HPC4AI). Meanwhile it will exploit advanced AI technologies, especially multi-objective reinforcement learning, to empower job scheduling and resource allocation in HPC (AI4HPC). The project aims to develop an intelligent workload-management framework named MINT in which distinctive computational resource requirements of hybrid workloads will be automatically identified and fulfilled to achieve extreme resource efficiency and satisfactory user experience. Key research thrusts are: understanding performance implications of diverse workloads on supercomputers via model-driven analysis; new intelligent multi-resource scheduling methods; smart resource-allocation strategies for minimal workload interference; and extensive evaluation of the proposed framework through trace-based simulation and testing. The deliverables include a new workload-management framework and open-source software releases for intelligent management of hybrid workloads on extreme-scale systems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.