Personalized FL with Communication Compression on TMLR
The paper Personalized Federated Learning with Communication Compression was accepted by the Transactions on Machine Learning Research.
The paper “Personalized Federated Learning with Communication Compression” was accepted by the Transactions on Machine Learning Research (TMLR), link to the paper: https://openreview.net/forum?id=dZugyhbNFY. TMLR is a venue for the dissemination of machine learning research which emphasizes technical correctness over subjective significance.
I was glad to work with my peers:
- Peter Richtárik from King Abdullah University of Science and Technology, KSA
- Aritra Dutta from Artificial Intelligence Initiative University of Central Florida, USA
- El Houcine Bergou from Mohammed VI Polytechnic University, Morocco
Accepted papers to TMLR can be browsed directly from OpenReview.
Our Work
The limited amount of data for training big models for Machine Learning can lead to poor machine learning models, or make even training impossible. Federated learning (FL) (research.google/pubs/pub45648, arXiv:1912.04977, arXiv:1908.07873) has emerged as an interdisciplinary field focused on addressing these issues by training machine learning models directly on edge devices or organizations. In contrast to training traditional machine learning (ML) models in data centers, Federated Learning trains ML models over local datasets contained on resource-constrained heterogeneous edge devices.
Existing FL algorithms mainly aim to learn a single global model for all participating devices. However, it may not be helpful to all devices participating in the training due to the heterogeneity of the data across the devices. Recently, Hanzely and Richtárik (2020) proposed a new formulation for training personalized FL models aimed at balancing the trade-off between the traditional global model and the local models that could be trained by individual devices using their private data only. They derived a new algorithm, called Loopless Gradient Descent (L2GD), to solve it and showed that this algorithm leads to improved communication complexity guarantees.
On our paper, we equipped their L2GD algorithm with a bidirectional compression mechanism to further reduce the communication bottleneck between the local devices and the server. Unlike other compression-based algorithms used in the FL setting, our Compressed L2GD algorithm operates on a probabilistic communication protocol, where communication does not happen on a fixed schedule. All our algorithms are equipped with the theory for convex and non-convex cases for finite sum minimization problems, which includes solving the Empirical Risk Minimization (ERM) fomulation.
Our Compressed L2GD algorithm maintains a similar convergence rate as vanilla SGD without compression. To validate the efficiency of our algorithm, we performed diverse numerical experiments and demonstrated the benefits of using our algorithm during training modern Deep Learning models used in Image Processing tasks:
We included experiments with Logistic Regression, the model that is widely used in Statistics, Optimization, and ML for training shallow models when the modeling effort is focused on feature design rather than creating complex family of function classes.
 |
 |
 |