Hierarchical Global Asynchronous Federated Learning Across Multi-Center

Wei Xie, Runqun Xiong, Junzhou LuoACML 2024 · OpenReview

TL;DR

This paper introduces HGA-FL, a novel hierarchical federated learning approach that tackles the challenges of training models across multiple geographically distributed centers. By combining asynchronous global aggregation with synchronous sub-center aggregation and local regularization, HGA-FL significantly improves training efficiency and model generalization, especially under conditions of data heterogeneity and network latency.

What Problem Does This Paper Address?

Federated Learning (FL) is a powerful technique for training machine learning models on decentralized data without sharing raw information. However, applying FL to real-world scenarios with geographically dispersed "regional centers" (or "multi-centers") presents significant challenges. These centers often have varying computational capabilities and network latencies, leading to "stragglers" – slower participants that delay the entire training process when synchronous aggregation is used. While asynchronous FL can alleviate these delays, it introduces its own set of problems, such as stale model parameters and outdated gradients, especially in hierarchical settings where aggregation happens at multiple levels. Existing hierarchical FL methods primarily focus on synchronous aggregation, which is inefficient in heterogeneous environments. This paper aims to bridge this gap by proposing a novel hierarchical asynchronous FL framework that balances efficiency and accuracy.

Methods

The core of this paper is the Hierarchical Global Asynchronous Federated Learning (HGA-FL) framework. It employs a three-tier architecture: a global center, multiple sub-centers, and individual clients within each sub-center. The key innovation lies in its two-level aggregation strategy:

  • Global Asynchronous Aggregation: The global center aggregates models from different sub-centers asynchronously. This means it doesn't wait for all sub-centers to report before proceeding, thus mitigating the straggler problem at the highest level. A global buffer is used to store incoming sub-center models.
  • Synchronous Sub-center Aggregation: Within each sub-center, clients' models are aggregated synchronously. This leverages the fact that clients within a single sub-center are often more homogeneous in terms of capabilities and latency.
  • Local Consistent Regularization: To further enhance model consistency and prevent divergence, HGA-FL incorporates local regularization techniques within each client's training process. This helps align local models with the global model.

The paper also provides a theoretical analysis of HGA-FL's convergence rate under non-convex optimization settings, demonstrating its stability.

Datasets

The paper evaluates HGA-FL on the following datasets:

  • EMNIST: A dataset of handwritten characters.
  • FashionMNIST (F-MNIST): A dataset of clothing item images.
  • CIFAR-10: A dataset of 10 common object classes.

For introducing data heterogeneity, they use:

  • Label Dirichlet Allocation (LDA): A method to draw samples from a Dirichlet distribution per client, controlling the degree of data imbalance.
  • Local Long-Tailed (LLT) partitioning: Another method to create imbalanced datasets.

Key Insights and Ideas

  • The "Best of Both Worlds" Approach: HGA-FL successfully combines the efficiency of asynchronous aggregation at the global level with the stability of synchronous aggregation at the sub-center level. This hybrid approach is crucial for handling the inherent heterogeneity of multi-center FL.
  • Addressing Staleness Effectively: By carefully modeling gradient and model staleness across different aggregation tiers, HGA-FL manages to maintain stable convergence, a significant challenge in asynchronous FL.
  • Theoretical Guarantees: The paper doesn't just propose a method; it provides theoretical convergence analysis, offering confidence in its performance under non-convex settings.
  • Impact of Hyperparameters: The study highlights the sensitivity of HGA-FL to hyperparameters like ng (global step rate) and α (local regularization parameter), showing that specific ranges lead to optimal performance.
  • Buffer Size Matters: A larger global buffer size (K) generally improves global test accuracy, suggesting that accumulating more sub-center updates before global aggregation can be beneficial.

Key Results

  • Superior Generalization: HGA-FL consistently outperforms baseline two-level aggregation methods in terms of global model generalization ability across various datasets (EMNIST, F-MNIST, CIFAR-10). For instance, on EMNIST with LDA αd=0.2, M=8, R=8, and K=7, HGA-FL achieved a test accuracy of 77.46%, significantly higher than other methods.
  • Robustness to Heterogeneity: The method demonstrates strong performance even under challenging conditions of data heterogeneity, latency, and gradient staleness.
  • Faster Convergence (in some aspects): While not always the fastest in terms of raw communication rounds, HGA-FL achieves better accuracy with fewer rounds or more stable convergence curves compared to some asynchronous baselines.
  • Effectiveness of Hybrid Approach: Experiments show that HGA-FL-G (global asynchronous) combined with synchronous sub-center methods (like S-Avg, S-Prox) yields better results than purely asynchronous or synchronous multi-center approaches.

Limitations and Future Work

  • Hyperparameter Tuning: While the paper identifies optimal ranges for ng and α, finding these optimal values might still require careful tuning in practice.
  • Impact of Sub-center Number (M): The study suggests that a larger number of sub-centers (M) with a fixed total number of clients might introduce more gradient variance and model drift, potentially reducing global test accuracy. This indicates a trade-off in scaling the number of sub-centers.
  • Future Work: The paper implicitly suggests further exploration into optimal hyperparameter selection strategies and understanding the scalability limits of the HGA-FL framework with a very large number of sub-centers. The theoretical analysis could also be extended to more complex non-convex settings or different types of data distributions.

Summary updated

Abstract

Federated learning for training machine learning models across geographically distributed regional centers is becoming prevalent. However, because of disparities in location, latency, and computational capabilities, synchronously aggregating models across different sites requires waiting for stragglers, leading to significant delays. Traditional asynchronous aggregation across regional centers still faces issues of stale model parameters and outdated gradients due to the hierarchical aggregation involving local clients within each center. To address this, we propose Hierarchical Global Asynchronous Federated Learning (HGA-FL), which combines global asynchronous model aggregation across regional centers with synchronous aggregation and local consistent regularization alignment within each local center. We theoretically analyze the convergence rate of our method under non-convex optimization settings, demonstrating its stable convergence during the aggregation. Experimental evaluations show that our approach outperforms other baseline two-level aggregation methods in terms of global model generalization ability, particularly under conditions of data heterogeneity, latency, and gradient staleness.
BibTeX
@inproceedings{xie2024hierarchical,
  author = {Wei Xie and Runqun Xiong and Junzhou Luo},
  title = {{Hierarchical Global Asynchronous Federated Learning Across Multi-Center}},
  booktitle = {ACML},
  year = {2024},
  url = {https://openreview.net/forum?id=MpXwb1SRWF},
}
Figure