Sure, here's a blog-style summary of the paper "FedPrune: Towards Inclusive Federated Learning":
TL;DR
This paper introduces FedPrune, a novel federated learning framework that addresses the performance degradation caused by client heterogeneity. By adaptively serving smaller model "subsets" to slower devices, FedPrune improves training efficiency, accuracy, and fairness, making federated learning more inclusive.
What Problem Does This Paper Address?
Federated learning (FL) allows training machine learning models across decentralized devices while preserving data privacy. However, FL's effectiveness is significantly hampered by two main challenges:
- System Heterogeneity: Devices have varying computational power, memory, and network bandwidth. This leads to "slow clients" or "stragglers" that take much longer to train.
- Statistical Heterogeneity: Data distribution across clients is often non-Independent and Identically Distributed (non-IID).
In traditional FL, like Federated Averaging (FedAvg), slow clients are often dropped from training rounds to maintain efficiency. This exclusion not only limits the overall learning process but also biases the model towards faster clients, leading to reduced test accuracy and unfairness. Waiting for slow clients, on the other hand, drastically increases training time and degrades user experience. The paper aims to solve this by making FL inclusive of all clients, regardless of their device capabilities.
Methods
FedPrune tackles the problem of client heterogeneity through a multi-pronged approach:
- Differential Model Serving (DMS): This is the core idea. Instead of serving the same large model to all clients, FedPrune serves different sized models based on client capabilities. Slow clients receive smaller "sub-models," while fast clients receive the full, larger model. This allows slow clients to complete training within their resource constraints.
- Sub-model Selection using Activations: To determine which parts of the model to prune for slow clients, FedPrune uses a data-driven approach. Initially, a random sub-model is used. After a few rounds, the system analyzes the post-activation values of neurons in the centralized model. Neurons with low or no activation are candidates for pruning, effectively creating a smaller, task-relevant sub-model for slow clients.
- Model Generalization using Central Limit Theorem (CLT): To improve robustness, especially with non-IID data, FedPrune employs an aggregation technique inspired by the Central Limit Theorem. Instead of simply averaging client model parameters, it calculates the mean and standard deviation of parameters across clients and then samples from a Normal distribution with these parameters. This helps generalize better, even when client data distributions differ significantly.
Datasets
The paper evaluates FedPrune on several real-world federated datasets:
- FEMNIST: A federated version of the extended MNIST dataset with 62 classes, used for image classification. Two versions were used: one less heterogeneous and a skewed non-IID version.
- FMNIST: The Fashion MNIST dataset, also used in previous FL research.
- CIFAR-10: A standard image classification dataset, adapted for federated learning with an IID distribution.
- Sent140: A text dataset of tweets used for sentiment analysis, employing an LSTM classifier.
The experiments utilize the LEAF benchmarking framework for large-scale simulations and a real testbed with mobile devices.
Key Insights and Ideas
- Adaptive Model Sizing is Key: The most significant insight is that tailoring model size to device capability is crucial for inclusive FL. This simple yet effective strategy directly addresses the bottleneck of slow clients.
- Activations as a Pruning Guide: Using post-activation values to identify less important neurons is an intelligent way to create task-specific sub-models without requiring prior knowledge of the data distribution.
- CLT for Robust Aggregation: Applying CLT principles to model aggregation offers a principled way to handle statistical heterogeneity, leading to more robust and generalized models.
- Fairness Gains from Inclusivity: By not dropping slow clients, FedPrune inherently improves fairness. The paper shows that this inclusivity also leads to better overall test accuracy and reduced variance in performance across clients.
Key Results
- Improved Training Time: On a slow mobile device (Nokia 1), FedPrune with a 50% pruned model reduced training time by approximately 66.7% compared to the large model, bringing it closer to the training time of a fast device.
- Enhanced Test Accuracy: FedPrune consistently achieved better test accuracy than FedAvg across various datasets and scenarios, especially under high system and statistical heterogeneity. For instance, in a scenario with 90% slow clients and skewed data, FedPrune showed a 22.7% test accuracy improvement over FedAvg.
- Better Convergence: Figures in the paper (e.g., Figure 6) demonstrate that FedPrune leads to faster convergence and higher final test accuracy compared to FedAvg, particularly when many clients are slow.
- Improved Fairness: FedPrune exhibited lower variance in test accuracy across clients compared to FedAvg, indicating a more equitable performance distribution.
Limitations and Future Work
- Two-Model Approach: The current FedPrune implementation uses a two-model approach (global for fast, sub-model for slow). The authors suggest that in the future, it would be beneficial to explore customizing model sizes for each client individually based on their specific characteristics.
- Synchronous FL Focus: The current work focuses on synchronous FL. The authors note that exploring asynchronous FL approaches for mitigating stragglers would be an interesting future direction, though they acknowledge the challenges with bounded-delay assumptions in asynchronous schemes.
- Tuneable Parameters: Parameters like the pruning rate (k%) and the model update round (r) are tuneable. Further research could explore more adaptive methods for setting these parameters.
Summary updated