CRII: III: Towards Improving the Handling of Heterogeneity and Personalization in Federated Learning

As awareness of the need for privacy preservation continues to grow in society, new legal restrictions, such as the General Data Protection Regulation (GDPR), are emerging. Such laws demand that businesses and organizations not share their clients' raw data for any commercial purposes. Federated Learning (FL) is a distributed machine learning paradigm that works with decentralized data while preserving privacy. FL has gained widespread interest and has been applied in numerous applications, such as healthcare, education, and intelligent manufacturing. However, FL faces some challenges that come from, executing on diverse types of data and devices, such as mobile phones and Internet of Things (IoT) devices. This project aims to address the aforementioned issues in heterogeneous FL by developing mathematical models and efficient algorithms. In addition, the project will integrate trustworthy ML research into new curriculum development and support students from underrepresented groups. Heterogeneous FL faces two significant challenges: (1) each client in FL may generate data according to a distinct distribution; (2) heterogeneous clients, such as mobile phones and IoT devices, are equipped with a wide range of computation and communication capabilities. To address these challenges, this project will dramatically push the boundary of knowledge via the following two integrated research thrusts: (i) The research team aims to tackle data heterogeneity in FL by designing two advanced personalized learning methods. Specifically, the proposed solutions aim to balance the generalization ability from the global model and the personalization ability from the local model, improving both the global model and personalized local models. (ii) The team will study heterogeneous neural network aggregation for FL by providing advanced memory-efficient local training strategies for small devices. In addition, the project will make use of mutual knowledge distillation to improve the generalization ability of the local models. Finally, the team proposes a unified FL framework that integrates data-free knowledge aggregation with advanced memory-efficient solutions to tackle both heterogeneity issues simultaneously. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.