Introduction
Standard Federated Learning (FL) aims to train a single global model across decentralized devices without sharing raw data. While effective for independent and identically distributed (IID) data, it often fails in real-world scenarios characterized by non-IID (heterogeneous) data. A "one-size-fits-all" model typically yields poor performance for individual clients with unique data distributions. Additionally, the massive size of modern deep learning models creates a significant communication bottleneck for edge devices. This paper introduces a framework that leverages model pruning not just for compression, but as a primary mechanism for personalization and efficiency.
Methodology: Sub-FedAvg and Hybrid Pruning
The authors propose identifying unique "subnetworks" for each client that best fit their local data through iterative pruning.
Hybrid Pruning Strategy
To balance flexibility and hardware efficiency, the framework employs a hybrid pruning approach:
- Unstructured Pruning: Applied to fully connected layers, removing individual weights based on magnitude.
- Structured (Channel) Pruning: Applied to convolutional layers, removing entire filters to physically shrink the model and speed up inference.
The Sub-FedAvg Algorithm
The core contribution is Sub-FedAvg. Unlike traditional FedAvg, which averages all parameters, Sub-FedAvg averages only the parameters present in the intersection of client subnetworks. Parameters pruned by most clients are considered task-specific or "personalized," while those retained across many clients are treated as "common" features. The algorithm uses "Mask Distance" (Hamming distance between binary masks) to determine when a subnetwork has stabilized and pruning should stop.
Key Insights: Heterogeneity as a Blessing
A central theme of this work is that data heterogeneity is a "blessing" for personalization. The unique local data distributions allow clients to find distinct, high-performing subnetworks. The authors observed that clients with similar data labels tend to retain similar parameters after pruning, justifying the intersection-based averaging in Sub-FedAvg. In this view, pruning is a form of "forgetting" irrelevant parts of the global model to better suit a specific local context.
Experimental Results
The method was evaluated on MNIST, EMNIST, CIFAR-10, and CIFAR-100 benchmarks modified for non-IID settings.
Accuracy and Convergence
Sub-FedAvg consistently outperformed standard baselines. On non-IID CIFAR-10, Sub-FedAvg (Unstructured) achieved 86.01% accuracy, compared to only 58.99% for standard FedAvg. Furthermore, the method reached target accuracy 2-10x faster than traditional approaches.
Efficiency Gains
- Communication: For CIFAR-10, communication costs dropped from 2.48 GB (FedAvg) to 1.39 GB (Hybrid pruning).
- Computation: Hybrid pruning led to a 2.4x reduction in FLOPs, facilitating faster on-device inference.
Conclusion and Future Directions
While Sub-FedAvg shows significant promise, it introduces hyperparameter sensitivity regarding pruning rates and stabilization thresholds. Future research could explore "similarity-aware" client selection to further optimize the aggregation process. Additionally, maintaining accuracy at extreme sparsity levels for complex datasets like CIFAR-100 remains an ongoing challenge.
Summary updated