TL;DR
This paper introduces FedCPD, a federated learning method that significantly reduces communication overhead by combining complementary pruning with device-to-device (D2D) communication. By intelligently pruning model parameters and enabling clients to share updates directly, FedCPD achieves substantial communication cost savings while maintaining or even improving model accuracy.
What Problem Does This Paper Address?
Federated learning (FL) allows devices to collaboratively train a shared model without centralizing their data. However, a major bottleneck in FL is the frequent and often redundant transmission of model parameters between clients and the central server. This communication overhead can be substantial, especially with large models and numerous devices. Existing model pruning techniques aim to reduce this by removing less important weights, but they often struggle to identify critical parameters, potentially harming model performance. Furthermore, the physical distance and network conditions between devices and the server (D2S) can lead to high communication latency.
Methods
The proposed FedCPD algorithm tackles these issues through two main innovations:
- Complementary Pruning: This technique involves applying pruning masks on both the server and clients. Crucially, FedCPD retains information about pruned parameter locations, preventing the loss of potentially important weights that might be discarded by simpler pruning methods.
- Device-to-Device (D2D) Chain Updates: Instead of every client sending updates directly to the server, FedCPD implements a chain-like update mechanism. Only the first client in a chain downloads the global model from the server. Subsequent clients in the chain receive the updated model from the previous client for their local training. This significantly reduces the frequency of direct client-server communication.
After the last client in a chain completes its training, its final model parameters are adjusted using the inverse pruning mask (via a Hadamard product) before being uploaded to the server. The server then aggregates these parameters with the initial sparse global model, effectively filling in the pruned locations and creating a new global model for the next round.
Datasets
The paper evaluates FedCPD on four standard image classification datasets:
- MNIST
- Fashion MNIST (FMNIST)
- CIFAR-10
- CIFAR-100
Key Insights and Ideas
- Pruning is not just about removal, but intelligent retention: The "complementary" aspect of pruning, which involves retaining information about pruned weights, is key to maintaining model accuracy. This suggests that simply removing small weights isn't always optimal.
- Decentralizing communication pays off: Shifting some communication from client-server (D2S) to client-client (D2D) dramatically reduces the load on the central server and can bypass D2S latency issues.
- Efficiency doesn't have to sacrifice accuracy: FedCPD demonstrates that significant reductions in communication overhead and computational load can be achieved without a corresponding drop in model performance, and in some cases, even improving it.
Key Results
The paper reports that FedCPD significantly reduces communication costs while improving model accuracy across the tested datasets. Specifically:
- Reduced Communication Overhead: The method substantially lowers the amount of data transmitted.
- Improved Model Accuracy: FedCPD consistently outperforms other algorithms in terms of test accuracy and exhibits lower loss values.
- Efficiency Gains: The algorithm requires lower communication volume and FLOPs (Floating Point Operations) compared to baseline methods. For example, on CIFAR-100, FedCPD achieves 79% accuracy with a communication overhead of 7.8E+07 Mb and 6.2 TFLOPs at round 90, indicating a favorable trade-off.
Limitations and Future Work
Not discussed in the paper.
Summary updated