A Multi-Rate Feedback Control Framework for Design and Analyzing of Decentralized and Federated Learning

Distributed systems hold great promise to realize scalable processing and real-time intelligence required for our modern life, revolutionizing how we interact with technology and enabling us to easily tackle complex problems. However, despite extensive research in distributed algorithms and systems, several challenges persist in their synthesis and application. In particular, the current algorithm design process is not scalable. Indeed one needs to design a new algorithm and develop the corresponding analysis specific to each application scenario and set of requirements. There has been an urgent need to unify various subclasses of distributed algorithms to provide insights and streamline the design and analysis. The framework developed in this project will benefit a wide range of applications beyond machine learning, such as control theory and signal processing. The proposed activities also offer rich opportunities for engaging undergraduate students in cross-disciplinary research and K-12 outreach activities. This proposal advocates a generic “model” of distributed algorithms by leveraging theory and techniques from stochastic multi-rate feedback control systems. Such a generic model can abstract important features of distributed algorithms, such as guaranteed differential privacy, compressed communication, or occasional communication, into tractable modules. Building upon these abstract models, we design a framework with superior modeling power to encompass a substantial class of distributed algorithms. Such a framework can be used to analyze the entire algorithm class, but more importantly, it helps streamline the design of new algorithms in the sense that features arising from different application domains can be easily integrated. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.