CSR: RI: Small: Sustainable Large Scale Machine Learning via Multi-Level Optimization on Algorithm, System, and Meta Learning

Environmental sustainability has become a critical global concern in both technological and economic spheres. Artificial Intelligence (AI), while addressing the environmental consequences of human activities, also contributes to environmental degradation due to its reliance on energy-intensive hardware like GPUs, which are used in diverse applications, ranging from chatbots (e.g., ChatGPT) to AI for Science applications (e.g., AlphaFold3). The project’s novelties are the introduction of a holistic approach combining system architecture, machine learning (ML), and software engineering to boost energy-efficiency, sustainable, and collaborative AI development. The project's broader significance and importance are its offering (i) easy-to-use, low-cost, scalable, and standardized open-source software to promote environmental sustainability in AI development; and (ii) multi-disciplinary education and research training opportunities across various educational levels. This project takes a holistic view of all levels of the AI development pipeline and seeks to harness recent progress in ML Systems, AutoML, and Collaborative Learning (CL) techniques to foster an environmentally sustainable approach to AI and LPM development across natural language processing, computer vision, and AI4Science domains. The project’s goal is to enhance training efficiency, reduce power consumption, improve hardware utilization, and expedite model development explorations. The project introduces a suite of measures, including efficient parallelization strategies, well-tuned hyperparameters with minimal trial rounds, and optimized hardware utilization. By aligning all aspects of the computing process with sustainability, a comprehensive approach to AI computing can be adopted, fostering a more responsible and environmentally mindful practice. The project’s objectives include (i) sustainable hardware utilization through optimized distributed ML strategies; (ii) efficient collaboration via cognitive scheduling and CL; (iii) learning to develop AI models and hyperparameters faster; and (iv) developing software toolkits for sustainable ML at all levels through multi-level optimization. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.