Distributed Optimization with Dimension-Free Communication: Theory and Algorithms

This project advances the scientific foundations of distributed optimization, a core technology that enables modern artificial intelligence (AI) systems to learn collaboratively across many computers or devices at once. As AI models grow from millions to billions of parameters, the exchange of information among participating devices has become a major barrier to scalability, energy efficiency, and practical deployment. Existing methods require this exchange to grow in proportion to model size, making distributed training increasingly costly and difficult in bandwidth-limited settings such as edge computing, mobile devices, and wireless networks. This project develops a fundamentally new approach in which participating devices exchange only single numerical values, making communication costs independent of model size. By reducing communication costs while maintaining learning performance, the research supports more scalable and efficient AI systems, with direct applications such as locally detecting privacy risks in mobile software and improving personalized product recommendations while keeping user data on local devices. The project also develops open-source software, simulation platforms, and experimental testbeds to accelerate research adoption by academia and industry. Educational activities integrate research outcomes into graduate curricula while expanding research opportunities for graduate, undergraduate, and K-12 students through mentoring, outreach, and hands-on experiences in distributed learning. Communication overhead that grows with model size remains a fundamental and unresolved bottleneck in distributed optimization, limiting the scalability of large-scale machine learning. The project studies how to achieve dimension-free communication, in which communication cost no longer depends on model size, while preserving convergence speed, robustness, and theoretical guarantees for large-scale nonconvex machine learning problems. The research investigates three tightly integrated thrusts. First, it develops a general scalar-only communication framework that enables distributed optimization algorithms to exchange only scalar information during each communication round, while establishing convergence, communication complexity, and sample complexity guarantees. Second, it incorporates efficient second-order curvature approximation information to accelerate optimization without sacrificing dimension-free communication, including methods based on diagonal and low-rank approximations. Third, it develops participation-aware optimization methods that remain robust when only a subset of computing devices participates in each communication round, reflecting realistic distributed environments with heterogeneous devices and intermittent connectivity. The proposed algorithms are analyzed theoretically and evaluated through software simulations, heterogeneous edge-computing testbeds, and large-scale wireless network emulation. The project also validates the methods through two representative applications: distributed fine-tuning of large language models for mobile software privacy analysis and for personalized recommendation. Together, these advances establish a new theoretical and algorithmic foundation for communication-efficient distributed optimization, enabling future AI systems to scale more efficiently across distributed computing environments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.