PerHeFed: A general framework of personalized federated learning for heterogeneous convolutional neural networks

Le Ma, Yuying Liao, Bin Zhou, Wen XiWWW 2023 · DOI

TL;DR

This paper introduces PerHeFed, a novel framework for personalized federated learning (FL) that allows devices to use heterogeneous model architectures. By enabling clients to design their own local models and share only specific sub-models, PerHeFed significantly reduces communication costs and enhances privacy while achieving competitive accuracy, even across different network types.

What Problem Does This Paper Address?

Traditional federated learning (FL) assumes all participating devices use models with the same structure. This uniformity is a major bottleneck in real-world applications, especially in the Internet of Things (IoT), where devices have vastly different hardware capabilities, network conditions, and data distributions. This heterogeneity makes it difficult to train a single global model that performs well for everyone. Furthermore, transmitting entire model parameters in FL raises concerns about communication overhead and privacy leakage. Existing personalized FL approaches often struggle with model heterogeneity, particularly when dealing with diverse convolutional neural network (CNN) architectures.

Methods

PerHeFed tackles model heterogeneity by allowing each client device to autonomously design its local model architecture and select a "shared sub-model" to communicate. The core of PerHeFed lies in its novel aggregation method for these heterogeneous sub-models.

  • Heterogeneous Sub-models: Clients can choose any combination of convolutional layers, or even sparse versions of their models, as their shared sub-model. This flexibility is key to accommodating diverse device capabilities and personalized needs.
  • Fit-all Global Model: The parameter server initializes a "fit-all" global model that is large enough to encompass the structures of all incoming sub-models.
  • Mapping Relations: The framework establishes mapping relations between each client's sub-model and the global model. This is crucial for aligning parameters from different structures. The paper distinguishes between:
  • Inter-layer heterogeneity: Differences in the number of convolutional layers.
  • Intra-layer heterogeneity: Differences in the number of neurons (filters) within a layer.
  • Personalized Sub-model Aggregation: The server aggregates the received sub-models using these mapping relations. This process is designed to be agnostic to the specific network architectures, allowing for flexible combinations.
  • Communication Efficiency: Only the selected shared sub-model parameters are transmitted, drastically reducing communication costs compared to sending entire models.
  • Privacy Enhancement: By sharing only sub-models, the risk of privacy leakage from complete model parameters is also reduced.

Datasets

The paper uses the following datasets for evaluation:

  • SVHN (Street View House Numbers): A real-world image dataset of house numbers, used for evaluating performance on non-IID data.
  • CIFAR-10: A standard image classification dataset with 10 labeled classes, also used for evaluating performance, including on non-IID data.

The authors also describe setting up "extreme non-IID" configurations for these datasets to rigorously test the framework's robustness.

Key Insights and Ideas

  • Model Heterogeneity is Essential for Real-World FL: The paper strongly argues that accommodating diverse model architectures is not just a convenience but a necessity for practical FL applications, driven by personalized data, hardware, and security needs.
  • Convolutional Layers are Aggregable: A key intuition is that convolutional layers, primarily responsible for feature extraction, share a common functional purpose across different CNN architectures. This makes their parameters aggregable, even if they come from different layers or have different structures.
  • Trade-off Between Heterogeneity and Accuracy: While PerHeFed excels at handling heterogeneity and reducing communication costs, there's an observed trade-off where increased model heterogeneity can sometimes lead to a slight decrease in global model accuracy. However, this sacrifice is often deemed acceptable given the benefits.
  • Flexibility is Paramount: The framework's strength lies in its ability to adapt to a wide range of heterogeneous scenarios, including cross-model FL, without requiring clients to compromise on their model design.

Key Results

  • Communication Cost Reduction: PerHeFed significantly reduces communication costs. For example, on SVHN, it compresses approximately 50% of the shared sub-model parameters compared to FLOP. In experiments, communication costs were shown to be 0.33 times (for ResNet34) and 0.42 times (for DenseNet121) that of conventional FL (FedAVG).
  • Accuracy Performance:
    • On SVHN, PerHeFed achieved a 4.38% drop in accuracy compared to FLOP.
  • On CIFAR-10, PerHeFed even achieved a 0.3% improvement in accuracy.
  • In extreme non-IID settings, the accuracy gap between PerHeFed and baselines was reduced, with communication costs halved on SVHN for a minimal accuracy drop.
  • Cross-Model Hybrid FL: The framework demonstrated feasibility in combining parameters from different CNN models (ResNet34 and DenseNet121), showing that convolutional layer parameters from different positions are indeed fusible.

Limitations and Future Work

  • Accuracy Trade-off: The paper acknowledges that increased model heterogeneity can introduce noise and slightly affect global model accuracy.
  • Adverse Effects of Widely Different Shared Models: The authors note that the variability of shared models can impact model accuracy and plan to investigate ways to mitigate these adverse effects.
  • Model Poisoning: The paper suggests exploring how to regulate client behavior to prevent model poisoning in FL.
  • Generalization to Other Architectures: While experiments focused on CNNs, the authors believe PerHeFed is applicable to other neural network architectures like RNNs and plan to study this further.
  • Further Improvements: Methods like model fine-tuning and data augmentation are suggested as potential avenues to further improve inference accuracy.

Summary updated

BibTeX
@article{ma2023perhefed,
  author = {Le Ma and Yuying Liao and Bin Zhou and Wen Xi},
  title = {{PerHeFed: A general framework of personalized federated learning for heterogeneous convolutional neural networks}},
  journal = {WWW},
  year = {2023},
  doi = {10.1007/s11280-022-01119-x},
}
Figure