Federated Reinforcement Learning: From Heterogeneous Environments to Offline Data

With the explosive growth of machine learning (ML) based artificial intelligence (AI), reinforcement learning (RL) has achieved significant success in solving online sequential decision-making problems for a wide range of applications. This project explores federated reinforcement learning (FRL) for achieving collaborative intelligence in online sequential decision-making in networked intelligent systems. It will advance the fundamental understandings and develop fast and efficient algorithms for FRL, while addressing its unique challenges. The research outcomes of this project have the potential to enable intelligent control and management of wireless and computer networks, and also support various emerging AI applications over networked systems, such as collaborative robotics, connected and autonomous vehicles, multi-user mixed reality. The proposed research is also integrated with education activities at the PI's institution for graduate, undergraduate, and K-12 students via curriculum development, research experiences, and outreach. This project explores FRL for achieving networked intelligence in online sequential decision-making. It studies two important and unique aspects of FRL that have been under-explored so far: 1) it explores FRL for agents with heterogeneous distributions of environments, which have intricate impacts on the learning goal of FRL as well as the design and analysis of the FRL algorithms; 2) it exploits offline data samples for agents, where heterogeneous data distribution shifts result in non-trivial challenges that need to be addressed for the FRL algorithms. This project will obtain fundamental understandings of FRL with heterogeneous environments and from offline data, as well as develop fast and efficient FRL algorithms for these settings. It investigates innovative cross-disciplinary research of AI/ML, networking, and distributed computing by studying FRL, and draws on the tools from ML, optimization, statistics, and algorithm design to develop provably good yet practical solutions. The project aims to provide useful insights of AI/ML and networking research for future networked intelligent systems based on data analytics. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.