Federated Multi-Phase Curriculum Learning to Synchronously Correlate User Heterogeneity

Mingjie Wang, Jianxiong Guo, Weijia JiaIEEE Trans. Artif. Intell 2024 · OpenReview

TL;DR

This paper introduces FedCL, a novel Federated Learning (FL) approach that tackles the challenge of data heterogeneity among users. By employing a multi-phase curriculum learning strategy, FedCL synchronizes the learning pace of local models, preventing the global model from drifting and leading to improved generalization performance with fewer communication rounds.

What Problem Does This Paper Address?

Federated Learning (FL) allows multiple decentralized devices to collaboratively train a machine learning model without sharing their raw data, preserving privacy. However, a major hurdle in FL is data heterogeneity, where each device's local dataset has a different distribution (Non-IID data). This often causes local models to diverge from the global model, leading to slow convergence and a degraded final model. Existing solutions, termed "asynchronous FL," try to mitigate this by aligning models either locally or after drift has occurred, which can be suboptimal. This paper aims to address this by proposing a synchronous FL method that actively aligns user learning paces before significant drift occurs.

Methods

The core of FedCL is its synchronous multi-phase curriculum learning approach. Here's a breakdown of the key components:

  • Local Curriculum Learning: Each client trains its local model and generates a "difficulty score" for each data sample. This score reflects how challenging a sample is for the current local model.
  • Global Curriculum Ensemble: The server aggregates these local difficulty scores. To protect user privacy, clients send Gaussian Mixture Model (GMM) parameters representing their difficulty distributions. The server then uses these GMMs to estimate a global difficulty distribution.
  • Generator for Data Synthesis: A generator model is trained on the server to produce synthetic data that aligns with both the global data distribution and the aggregated difficulty distribution. This synthetic data helps to regularize local training and prevent overfitting.
  • Multi-Phase Synchronization: The global curriculum is divided into multiple phases, each defined by a difficulty threshold. Clients are "frozen" (their models are not updated) once they reach a certain level of synchronization with the current global phase. This ensures that all active clients are learning at a similar pace and difficulty level, effectively synchronizing their learning. The process moves to the next phase when all clients are frozen.
  • Model Aggregation: The server aggregates models from both frozen and unfrozen clients to maintain a robust global model.

This approach contrasts with asynchronous methods by actively scheduling and aligning user learning paces, aiming to approximate the behavior of centralized deep learning on homogeneous data.

Datasets

The paper evaluates FedCL on three real-world datasets:

  • MNIST: A dataset of handwritten digits, used for character classification.
  • CELEBA: A dataset of celebrity faces, used for binary classification (e.g., predicting if a celebrity is smiling).
  • EMNIST: An extension of MNIST that includes handwritten letters, also used for character classification.

The authors also explore data heterogeneity by using the Dirichlet distribution to create Non-IID data splits, where a smaller alpha value indicates higher heterogeneity.

Key Insights and Ideas

  • Synchronous Learning is Key: The paper strongly argues that synchronizing learning paces before model drift is crucial for effective FL, especially under Non-IID conditions. This is a departure from many existing asynchronous approaches.
  • Curriculum Learning for FL: Adapting curriculum learning principles to FL allows for a dynamic, data-aware difficulty assessment that can guide and align client learning.
  • Privacy-Preserving Difficulty Sharing: Using GMMs to represent local difficulty distributions is an innovative way to share information about data characteristics without revealing raw data.
  • Generator as a Regularizer: The synthetic data generated by FedCL's generator acts as a powerful regularizer, helping local models generalize better and align with the global objective.
  • Phased Synchronization for Robustness: The multi-phase approach allows for gradual synchronization, ensuring that models are not forced into a single learning state too quickly, which could be detrimental.

Key Results

  • Superior Generalization: FedCL consistently outperforms state-of-the-art baselines (FedAvg, FedProx, FedEnsemble, FedDistill, FedGen) in terms of top-1 test accuracy across MNIST, EMNIST, and CELEBA datasets, especially under severe data heterogeneity.
  • Faster Convergence: FedCL achieves optimized performance in significantly fewer communication rounds compared to other methods. For instance, it reduces rounds by 4.3 on CELEBA and 2.5 on MNIST on average compared to FedGen.
  • Robustness to Heterogeneity: The performance gains of FedCL become more pronounced as data heterogeneity increases (smaller alpha values).
  • Efficiency: The approach is designed to be efficient, with the generator training happening on the server and not imposing additional costs on clients.

Limitations and Future Work

  • Trade-off Analysis: The authors suggest future research could explore the trade-off between privacy and utility in synchronous FL.
  • Integration of Learning Progress: Further integration of learning progress for guaranteed data privacy is also mentioned as a future direction.
  • Hyperparameter Sensitivity: While the paper demonstrates robustness to hyperparameters, further investigation into optimal settings for diverse scenarios might be beneficial.

The paper concludes that FedCL is a promising solution for tackling user heterogeneity in FL, offering improved generalization and efficiency.

Summary updated

Abstract

Federated learning (FL) is a decentralized learning method used to train machine learning algorithms. In FL, a global model iteratively collects the parameters of local models without accessing their local data. However, a significant challenge in FL is handling the heterogeneity of local data distribution, which often results in a drifted global model that is difficult to converge. To address this issue, current methods employ different strategies, such as knowledge distillation, weighted model aggregation, and multitask learning. These approaches are referred to as asynchronous FL, as they align user models either locally or posthoc, where model drift has already occurred or has been underestimated. In this article, we propose an active and synchronous correlation approach to address the challenge of user heterogeneity in FL. Specifically, our approach aims to approximate FL as standard deep learning by actively and synchronously scheduling user learning pace in each round with a dynamic multiphase curriculum. A global curriculum is formed by an autoregressive autoencoder that integrates all user curricula on the server. This global curriculum is then divided into multiple phases and broadcast to users to measure and align the domain-agnostic learning pace. Empirical studies demonstrate that our approach outperforms existing asynchronous approaches in terms of generalization performance, even in the presence of severe user heterogeneity.
BibTeX
@article{wang2024federated,
  author = {Mingjie Wang and Jianxiong Guo and Weijia Jia},
  title = {{Federated Multi-Phase Curriculum Learning to Synchronously Correlate User Heterogeneity}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2024},
  url = {https://openreview.net/forum?id=oyINep1pay},
}
Figure