Loss Tolerant Federated Learning

Pengyuan Zhou, Pei Fang, Pan HuiarXiv 2021 · arXiv

Abstract

Standard Federated Learning (FL) often excludes devices with poor network connections to maintain system efficiency, leading to biased models that underrepresent certain demographics. This paper proposes ThrowRightAway (TRA), a loss-tolerant FL scheme that allows low-bandwidth devices to participate by intentionally ignoring some packet losses. By using adaptive aggregation to compensate for missing data, TRA ensures fairer participation and significantly improves global model performance in heterogeneous network environments.

Introduction

Federated Learning (FL) enables collaborative model training across decentralized devices while preserving data privacy. However, the practical deployment of FL faces significant challenges due to network heterogeneity. Mobile and IoT devices often have limited upload speeds or unreliable connections. To prevent these "stragglers" from delaying the training process, current FL systems typically employ threshold-based client selection, excluding any device that fails to meet a minimum performance requirement.

This exclusion creates a "never-represented" class of users. If networking conditions correlate with specific user demographics or geographic locations, the resulting global model becomes biased and performs poorly for the excluded groups.

The ThrowRightAway (TRA) Scheme

The researchers introduce ThrowRightAway (TRA), a lightweight approach designed to integrate seamlessly with existing FL algorithms. Rather than a binary participation threshold, TRA categorizes clients based on their reported network capacity and handles them adaptively.

  1. Categorization: Clients are labeled as "sufficient" or "insufficient" based on their network speed.
  2. Selection: The server selects a subset of clients for training rounds without discriminating against the "insufficient" group.
  3. Adaptive Communication:
    • Sufficient clients: The server requests retransmissions for any lost packets to ensure data integrity.
  4. Insufficient clients: To avoid bottlenecks, the server "throws away" lost packets and sets the corresponding values to zero.
  5. Adaptive Aggregation: The server compensates for the "thrown away" data by reweighting contributions from insufficient clients based on their packet drop rate ($r$).
The TRA scheme enables participation from all clients by using retransmissions for fast devices and "setting to zero" for slow ones to maintain training flow.

Experimental Setup

The study evaluated TRA using:

  • FCC Mobile Broadband Dataset (2019): Real-world traces of upload speeds and packet loss. Analysis showed that standard FL settings would exclude approximately 24% of users who have upload speeds below 2 Mbps.
  • Synthetic Datasets: Used to simulate varying degrees of statistical heterogeneity (non-IID data) to test the robustness of the aggregation method.

Key Results and Insights

The primary insight is that diversity is more important than data perfection. While threshold-based selection ensures high-quality updates from a few clients, it introduces a "data shift" that harms the global model.

Test accuracy significantly decreases as the percentage of eligible clients drops, highlighting the bias introduced by threshold-based selection.

As shown in the figure above, excluding even 10-30% of clients leads to a noticeable drop in final test accuracy. TRA mitigates this by allowing these clients to contribute, even if their updates are partially lossy.

  • Accuracy Gains: When paired with fairness-aware aggregation (q-FedAvg), TRA improved model accuracy by up to 10.35% in scenarios where 30% of clients were previously excluded.
  • Fairness: TRA reduced the variance in accuracy across different clients (from 1439 to 1031 in synthetic tests), indicating a more equitable model.
  • Personalization Trade-off: There is a minor trade-off; while the global model improves significantly, some individual clients might see a slight (~1%) decrease in their personalized model accuracy.

Conclusion

ThrowRightAway (TRA) demonstrates that embracing "lossy" updates is a viable strategy for building fairer and more accurate federated models. By shifting the focus from excluding stragglers to accommodating them, TRA addresses a critical source of bias in real-world FL deployments. Future work will focus on deeper theoretical convergence analysis and refining the integration with personalization techniques.

Summary updated

Abstract

Federated learning has attracted attention in recent years for collaboratively training data on distributed devices with privacy-preservation. The limited network capacity of mobile and IoT devices has been seen as one of the major challenges for cross-device federated learning. Recent solutions have been focusing on threshold-based client selection schemes to guarantee the communication efficiency. However, we find this approach can cause biased client selection and results in deteriorated performance. Moreover, we find that the challenge of network limit may be overstated in some cases and the packet loss is not always harmful. In this paper, we explore the loss tolerant federated learning (LT-FL) in terms of aggregation, fairness, and personalization. We use ThrowRightAway (TRA) to accelerate the data uploading for low-bandwidth-devices by intentionally ignoring some packet losses. The results suggest that, with proper integration, TRA and other algorithms can together guarantee the personalization and fairness performance in the face of packet loss below a certain fraction (10%-30%).
BibTeX
@misc{zhou2021loss,
  author = {Pengyuan Zhou and Pei Fang and Pan Hui},
  title = {{Loss Tolerant Federated Learning}},
  year = {2021},
  eprint = {2105.03591},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.03591},
}
Figure