H-FL: A Hierarchical Communication-Efficient and Privacy-Protected Architecture for Federated Learning

He YangIJCAI 2021 · DOI

TL;DR

This paper introduces H-FL, a novel hierarchical federated learning architecture designed to simultaneously optimize model performance, communication efficiency, and privacy. By introducing mediators and a runtime distribution reconstruction strategy, H-FL effectively tackles data heterogeneity, while a compression-correction mechanism reduces communication overhead, and differential privacy is integrated for enhanced privacy guarantees.

What Problem Does This Paper Address?

Federated Learning (FL) aims to train machine learning models collaboratively across decentralized devices without sharing raw data. However, achieving a good balance between three critical aspects – model performance, communication efficiency, and privacy guarantees – is a significant challenge. Traditional FL methods often make trade-offs, for instance, improving privacy might increase communication overhead or slow down convergence, thus requiring more communication rounds. This paper addresses the inherent contradiction that arises when trying to simultaneously satisfy all three requirements, particularly the tension between communication overhead and privacy when maintaining model performance.

Methods

H-FL introduces a hierarchical architecture with two main innovations:

  • Runtime Distribution Reconstruction: To combat the performance degradation caused by non-IID (non-independently and identically distributed) data across clients, H-FL employs a strategy to reconstruct local data distributions. Clients are clustered based on their local data distribution's entropy and KL divergence from a uniform distribution. Mediators are then assigned groups of clients, and these mediators help rearrange local training to create more uniform "virtual" distributions. This process aims to make the training problem closer to an IID scenario.
  • Compression-Correction Mechanism: To reduce communication overhead, H-FL uses a lossy compression technique on extracted features from the shallow model. This compression is based on Singular Value Decomposition (SVD) to create a low-rank representation. Crucially, a "bias corrector" is introduced to compensate for potential accuracy loss introduced by this compression, effectively building a differentiable mapping and improving the approximation of gradients.
  • Differential Privacy: Differential privacy is integrated into the local training process of each client, injecting noise into a portion of the model to protect individual data.

Datasets

The paper evaluates H-FL on two standard image recognition datasets:

  • FMNIST (Fashion-MNIST): A dataset of Zalando's article images, used with a modified LeNet5 network.
  • CIFAR10: A dataset of 60,000 32x32 color images in 10 classes, used with a modified VGG16 network.

Key Insights and Ideas

  • Holistic Approach to FL Trade-offs: H-FL is presented as the first attempt to holistically address model performance, communication overhead, and privacy simultaneously, rather than making pairwise trade-offs.
  • Mediators as Distribution Managers: The introduction of mediators is key to managing client clusters and orchestrating the runtime distribution reconstruction, effectively mitigating the impact of non-IID data.
  • Compression with Correction is Crucial: Simply compressing gradients or features can degrade performance. The "bias corrector" is vital for recovering accuracy lost during compression, demonstrating that effective communication reduction requires more than just aggressive compression.
  • Privacy Integration is Seamless: Differential privacy is applied during local training, suggesting that privacy can be integrated without fundamentally breaking the FL process.

Key Results

  • State-of-the-Art Performance: H-FL consistently outperforms FedAVG, DGC, and STC on both FMNIST and CIFAR10 datasets in terms of accuracy and convergence speed.
  • Improved Accuracy: On FMNIST, H-FL achieved 88.16% accuracy compared to FedAVG (82.28%), DGC (82.00%), and STC (82.12%). On CIFAR10, H-FL achieved 87.28% accuracy, significantly outperforming FedAVG (73.83%), DGC (81.25%), and STC (81.24%).
  • Reduced Communication: While not explicitly quantified in GBs for the final model, the improved convergence speed implies fewer communication rounds are needed to reach a target accuracy, thus reducing overall communication overhead. Figure 3(b) and 3(c) visually support this, showing H-FL requiring fewer rounds to reach target accuracies.
  • Bias Corrector Impact: The bias corrector significantly improves convergence speed and final accuracy, yielding a 2.47 percentage point improvement on CIFAR10.

Limitations and Future Work

  • Parameter Sensitivity: Figure 2 shows that H-FL's performance is influenced by parameters like sampling probability (P), sample size (S), compression ratio (C), and noise level (σ). Higher noise levels (σ) lead to slower convergence and worse performance.
  • Computational Overhead: While communication overhead is reduced, the introduction of mediators and the compression-correction mechanism might introduce additional computational overhead on clients and mediators, which is not explicitly detailed.
  • Future Work: Not explicitly discussed in the paper, but potential future work could involve exploring more sophisticated clustering and mediator assignment strategies, adaptive compression techniques, or investigating the trade-offs with different privacy mechanisms.

Summary updated

BibTeX
@inproceedings{yang2021h,
  author = {He Yang},
  title = {{H-FL: A Hierarchical Communication-Efficient and Privacy-Protected Architecture for Federated Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/67},
}
Figure