Federated Learning with Packet Losses

Angelo Rodio, Giovanni Neglia, Fabio Busacca, Stefano Mangione, Sergio Palazzo, Francesco Restuccia, Ilenia TinnirelloWPMC 2023 · OpenReview

TL;DR

This paper introduces UPGA-PL, a novel Federated Learning (FL) algorithm designed for wireless networks prone to packet losses. UPGA-PL effectively achieves optimal model convergence even with asymmetric loss probabilities between clients and the server, outperforming existing methods and matching ideal lossless performance.

What Problem Does This Paper Address?

Federated Learning (FL) enables distributed training of machine learning models across multiple devices (clients) coordinated by a central server. However, when FL is deployed over wireless networks, packet losses are a significant challenge. These losses can disrupt the communication between clients and the server, leading to slower convergence, biased models, and ultimately, a suboptimal final model. While previous research has focused on mitigating packet losses, this paper questions whether optimal convergence is achievable despite these losses, proposing a new aggregation strategy to tackle this fundamental issue.

Methods

The core of this paper's contribution lies in its novel aggregation strategy, UPGA-PL (Unbiased Pseudo-Gradient Aggregation with Packet Losses). Unlike traditional FL algorithms like FedAvg that average client models, UPGA-PL aggregates pseudo-gradients derived from client updates. This approach is combined with a mechanism to account for varying packet loss probabilities across clients. Specifically, UPGA-PL uses weights that compensate for clients experiencing more severe losses, effectively "unbiasing" the aggregation. This strategy aims to filter out noise introduced by packet losses and ensure convergence to the true optimal model, maintaining the same computational and communication complexity as FedAvg.

Datasets

The proposed UPGA-PL algorithm was evaluated on two datasets:

  • LEAF Synthetic Dataset: Used for multinomial classification, this dataset allows for direct control over statistical heterogeneity through parameters, making it suitable for testing theoretical assumptions.
  • MNIST Dataset: A real-world dataset for handwritten digit recognition, used with a CNN architecture to evaluate performance on a more complex, non-convex objective function.

Key Insights and Ideas

  • Optimal Convergence is Achievable with Losses: The paper's central insight is that FL can indeed converge to the optimal model even in the presence of significant and asymmetric packet losses, challenging prior assumptions that losses inherently introduce irreducible errors.
  • Pseudo-Gradients are Key: Aggregating pseudo-gradients, rather than raw models, provides a more robust mechanism to handle the noise and bias introduced by packet losses.
  • Unbiasing for Heterogeneous Losses: Explicitly accounting for differing client loss probabilities during aggregation is crucial for achieving unbiased and accurate convergence.
  • Trade-off Between Convergence Rate and Residual Error: While some methods might offer faster initial convergence but leave a residual error, UPGA-PL sacrifices some speed for guaranteed convergence to the true optimum.

Key Results

  • Performance Improvement: UPGA-PL significantly outperforms existing state-of-the-art solutions for FL under lossy channels, achieving 5-10 percentage points higher test accuracy on both the Synthetic LEAF and MNIST datasets.
  • Near-Lossless Performance: Remarkably, UPGA-PL matches the performance of FedAvg in ideal lossless scenarios after fewer than 150 communication rounds.
  • Robustness to Severe Losses: Even with high packet loss rates (e.g., 90% for some clients), UPGA-PL demonstrates robust convergence, unlike other methods that suffer from non-vanishing errors or noisy convergence.
  • Theoretical Guarantees: The paper provides theoretical analysis proving UPGA-PL's convergence to the optimal model under lossy channels, with a convergence rate comparable to FedAvg in lossless settings.

Limitations and Future Work

  • Assumptions: The theoretical analysis relies on standard assumptions for convex optimization and federated learning, including L-smoothness, µ-strong convexity, bounded gradient variance, and independent client outcomes. The paper acknowledges that spatial or temporal correlations in packet losses are not considered.
  • Future Directions: The authors suggest exploring scenarios where only parts of transmitted models are lost, investigating correlated losses (due to inter-channel interference or fading), and analyzing how performance changes when loss probabilities vary over time. They also note that while UPGA-PL achieves sublinear convergence, further research could explore methods that balance faster convergence with residual error mitigation.

Summary updated

Abstract

This paper tackles the problem of training Federated Learning (FL) algorithms over real-world wireless networks with packet losses. Lossy communication channels between the orchestrating server and the clients affect the convergence of FL training as well as the quality of the learned model. Although many previous works investigated how to mitigate the adverse effects of packet losses, this paper demonstrates that FL algorithms over asymmetric lossy channels can still learn the optimal model, the same model that would have been trained in a lossless scenario by classic FL algorithms like FedAvg. Convergence to the optimum only requires slight changes to FedAvg: i) while FedAvg computes a new global model by averaging the received clients' models, our algorithm, UPGA-PL, updates the global model by a pseudo-gradient step; ii) UPGA-PL accounts for the potentially heterogeneous packet losses experienced by the clients to unbias the pseudo-gradient step. Still, UPGA-PL maintains the same computational and communication complexity as FedAvg. In our experiments, UPGA-PL not only outperforms existing state-of-the-art solutions for lossy channels (by more than 5 percentage points on test accuracy) but also matches FedAvg's performance in lossless scenarios after less than 150 communication rounds.
BibTeX
@inproceedings{rodio2023federated,
  author = {Angelo Rodio and Giovanni Neglia and Fabio Busacca and Stefano Mangione and Sergio Palazzo and Francesco Restuccia and Ilenia Tinnirello},
  title = {{Federated Learning with Packet Losses}},
  booktitle = {WPMC},
  year = {2023},
  url = {https://openreview.net/forum?id=Xq7WdfAlyR},
}

Loading related content...

Figure