Friends to Help: Saving Federated Learning from Client Dropout

Heqiang Wang, Jie Xu

Introduction

Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models across distributed devices while maintaining data privacy. In an ideal FL setting, a central server coordinates a global model by aggregating updates from a swarm of participating clients. However, real-world deployments frequently encounter "passive client dropout." Unlike intentional client sampling—where a server selects a subset of devices to manage bandwidth—passive dropout occurs when clients unexpectedly vanish mid-training due to battery depletion, poor connectivity, or hardware failures. This paper introduces FL-FDMS (Federated Learning with Friend Discovery and Model Substitution), a novel framework designed to mitigate the bias and convergence issues caused by these unreliable network conditions.

The Challenge of Passive Dropout

Passive dropout is rarely uniform. Because dropout is often tied to external factors like signal strength or device age, it can introduce significant bias into the global model. For example, if a specific demographic of users consistently experiences poor connectivity, their unique data distributions will be underrepresented, leading to a model that fails to generalize to those users.

Existing strategies to handle this problem typically fall into two categories: ignoring the missing updates ("Dropout" method) or using "stale" updates from previous rounds ("Stale" method). Ignoring clients leads to the aforementioned bias, while using stale updates—often referred to as "zombie" updates—can be counterproductive. As the global model evolves, an update from several rounds ago becomes increasingly irrelevant, potentially steering the model away from the optimal convergence path.

The FL-FDMS Framework

The authors propose a strategy centered on Friend Model Substitution. The intuition is straightforward: if a client drops out, the server should substitute their missing update with a current update from a "friend"—another active client with a highly similar data distribution. The FL-FDMS algorithm operates in three distinct phases:

1. Friend Discovery

The server identifies relationships between clients by analyzing the "direction" of their model updates. By calculating the normalized cosine similarity of gradients over multiple rounds, the server builds a historical profile of client similarities. This allows the system to learn which clients are "friends" without ever needing to inspect their raw local data.

2. Model Substitution

When a client fails to report back during a training round, the server identifies the most similar active client (the "best friend") from the current pool. The update from this active friend is then used as a proxy for the missing client. This ensures that the specific data distribution represented by the dropped client is still accounted for in the global aggregation, albeit through a substitute.

3. Complexity Reduction via Hoeffding Races

A naive implementation of friend discovery requires calculating pairwise similarities for all clients, resulting in $O(K^2)$ complexity. To ensure the system scales to thousands of devices, the authors employ a mechanism inspired by Hoeffding races. Each client maintains a "candidate friend set" that is aggressively pruned over time. As statistical evidence mounts that certain clients are not similar, they are removed from the candidate set, drastically reducing the computational overhead on the server.

Experimental Evaluation

The researchers evaluated FL-FDMS using the MNIST and CIFAR-10 computer vision benchmarks. These datasets were tested in both clustered and general non-IID (Independent and Identically Distributed) settings to simulate various levels of data heterogeneity among users.

Key Findings

  • Superior Accuracy: FL-FDMS consistently outperformed both the Dropout and Stale baselines. In many scenarios, the performance of FL-FDMS approached the "Full Participation" ideal—a theoretical upper bound where no clients ever drop out.
  • Resilience to High Dropout: The algorithm proved remarkably robust. Even when 70% of clients dropped out in each round, FL-FDMS maintained high test accuracy and stable convergence, whereas traditional methods suffered from significant performance fluctuations.
  • Scaling to Complexity: The performance gains were more pronounced on the CIFAR-10 dataset than on MNIST. This suggests that as the learning task becomes more difficult and the data distributions more complex, the value of finding accurate "friend" substitutes increases.

Theoretical Insights

The authors provide a mathematical foundation for their approach, proving that FL convergence is fundamentally limited by "substitution error"—the distance between a missing update and its replacement. By using current updates from similar clients rather than outdated updates from the same client, FL-FDMS effectively minimizes this error, providing a tighter bound on convergence than stale-update methods.

Limitations and Future Directions

While FL-FDMS offers a significant leap forward, it relies on the assumption that every client has at least one active "friend" in any given round. If an entire cluster of similar clients drops out simultaneously (e.g., a regional power outage), the substitution mechanism lacks a valid proxy. Additionally, while the method preserves standard FL privacy, the calculation of pairwise similarities introduces new nuances that may require further protection through Differential Privacy or Secure Multi-Party Computation in future iterations.

Conclusion

FL-FDMS addresses a critical bottleneck in the practical deployment of Federated Learning. By leveraging the latent relationships between clients, the framework transforms passive dropout from a source of bias into a manageable challenge, ensuring that models remain accurate and inclusive even in the most unreliable network environments.

Summary updated

Abstract

Federated learning (FL) is a new distributed machine learning framework known for its benefits on data privacy and communication efficiency. Since full client participation in many cases is infeasible due to constrained resources, partial participation FL algorithms have been investigated that proactively select/sample a subset of clients, aiming to achieve learning performance close to the full participation case. This paper studies a passive partial client participation scenario that is much less well understood, where partial participation is a result of external events, namely client dropout, rather than a decision of the FL algorithm. We cast FL with client dropout as a special case of a larger class of FL problems where clients can submit substitute (possibly inaccurate) local model updates. Based on our convergence analysis, we develop a new algorithm FL-FDMS that discovers friends of clients (i.e., clients whose data distributions are similar) on-the-fly and uses friends' local updates as substitutes for the dropout clients, thereby reducing the substitution error and improving the convergence performance. A complexity reduction mechanism is also incorporated into FL-FDMS, making it both theoretically sound and practically useful. Experiments on MNIST and CIFAR-10 confirmed the superior performance of FL-FDMS in handling client dropout in FL.
BibTeX
@misc{wang2022friends,
  author = {Heqiang Wang and Jie Xu},
  title = {{Friends to Help: Saving Federated Learning from Client Dropout}},
  year = {2022},
  eprint = {2205.13222},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.13222},
}
Figure