NeTS:Small: Efficient Collective Communication for Distributed ML in the Cloud

Machine learning (ML) has transformed how we solve complex problems, from understanding languages to making accurate predictions in medicine and economics. However, modern ML models have grown extremely large—often involving trillions of parameters—that they can no longer run efficiently on a single computer. Instead, these enormous models must be distributed across many powerful processors, known as accelerators, in data centers. A critical challenge in running distributed ML models efficiently is managing the communication between accelerators. When different accelerators share information, this process — called collective communication — becomes a bottleneck, slowing down training and inference tasks. Current approaches to managing communication assume all connections between accelerators are equal. But in reality, connections can vary widely in speed and capacity, creating inefficiencies. This project aims to significantly improve collective communication by creating software tools and algorithms specifically designed for the diverse connections found in modern cloud-based accelerator systems. First, the project will measure how communication speeds and delays vary between accelerators, accounting for complexities like proprietary technologies and hidden network paths within data centers. Next, these measurements will be used to automatically generate optimized collective communication strategies tailored to specific cloud setups. This approach ensures that each deployment — whether it involves multiple servers within one data center, or servers spread across multiple data centers — benefits from customized, efficient collective communication. The project will have broader impacts beyond just technical improvements. Accelerating ML processing helps reduce the energy consumption and operational costs associated with data centers. The project will also make powerful ML tools more accessible for classroom education and engage with high school students through educational workshops to foster interest in science and technology. Releasing software tools, datasets, and findings openly will drive widespread adoption and enable the broader community to benefit from more efficient ML technologies, benefiting various fields like management and economics. All research artifacts developed through this project will be made publicly available via a dedicated project website at www.cccl.network. The website will be actively maintained throughout the project duration and archived for continued access beyond the project’s conclusion. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.