Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity

Saeed Vahidian, Mahdi Morafah, Bill Lin

Introduction

Standard Federated Learning (FL) aims to train a single global model across decentralized devices without sharing raw data. While effective for independent and identically distributed (IID) data, it often fails in real-world scenarios characterized by non-IID (heterogeneous) data. A "one-size-fits-all" model typically yields poor performance for individual clients with unique data distributions. Additionally, the massive size of modern deep learning models creates a significant communication bottleneck for edge devices. This paper introduces a framework that leverages model pruning not just for compression, but as a primary mechanism for personalization and efficiency.

Methodology: Sub-FedAvg and Hybrid Pruning

The authors propose identifying unique "subnetworks" for each client that best fit their local data through iterative pruning.

Hybrid Pruning Strategy

To balance flexibility and hardware efficiency, the framework employs a hybrid pruning approach:

  • Unstructured Pruning: Applied to fully connected layers, removing individual weights based on magnitude.
  • Structured (Channel) Pruning: Applied to convolutional layers, removing entire filters to physically shrink the model and speed up inference.

The Sub-FedAvg Algorithm

The core contribution is Sub-FedAvg. Unlike traditional FedAvg, which averages all parameters, Sub-FedAvg averages only the parameters present in the intersection of client subnetworks. Parameters pruned by most clients are considered task-specific or "personalized," while those retained across many clients are treated as "common" features. The algorithm uses "Mask Distance" (Hamming distance between binary masks) to determine when a subnetwork has stabilized and pruning should stop.

Local accuracy of individual clients as a function of pruning percentage, demonstrating the 'sweet spot' for personalized subnetworks.

Key Insights: Heterogeneity as a Blessing

A central theme of this work is that data heterogeneity is a "blessing" for personalization. The unique local data distributions allow clients to find distinct, high-performing subnetworks. The authors observed that clients with similar data labels tend to retain similar parameters after pruning, justifying the intersection-based averaging in Sub-FedAvg. In this view, pruning is a form of "forgetting" irrelevant parts of the global model to better suit a specific local context.

Experimental Results

The method was evaluated on MNIST, EMNIST, CIFAR-10, and CIFAR-100 benchmarks modified for non-IID settings.

Accuracy and Convergence

Sub-FedAvg consistently outperformed standard baselines. On non-IID CIFAR-10, Sub-FedAvg (Unstructured) achieved 86.01% accuracy, compared to only 58.99% for standard FedAvg. Furthermore, the method reached target accuracy 2-10x faster than traditional approaches.

Accuracy vs. Communication Rounds across CIFAR-10, EMNIST, and MNIST datasets, comparing Sub-FedAvg with standard baselines.

Efficiency Gains

  • Communication: For CIFAR-10, communication costs dropped from 2.48 GB (FedAvg) to 1.39 GB (Hybrid pruning).
  • Computation: Hybrid pruning led to a 2.4x reduction in FLOPs, facilitating faster on-device inference.

Conclusion and Future Directions

While Sub-FedAvg shows significant promise, it introduces hyperparameter sensitivity regarding pruning rates and stabilization thresholds. Future research could explore "similarity-aware" client selection to further optimize the aggregation process. Additionally, maintaining accuracy at extreme sparsity levels for complex datasets like CIFAR-100 remains an ongoing challenge.

Summary updated

BibTeX
@inproceedings{vahidian2021personalized,
  author = {Saeed Vahidian and Mahdi Morafah and Bill Lin},
  title = {{Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity}},
  booktitle = {Workshops @ ICDCS},
  year = {2021},
  doi = {10.1109/icdcsw53096.2021.00012},
}
Figure