Abstract
Federated Learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit its popularity, it has been observed that Federated Learning yields suboptimal results if the local clients' data distributions diverge. To address this issue, we present Clustered Federated Learning (CFL), a novel Federated Multi-Task Learning (FMTL) framework, which exploits geometric properties of the FL loss surface, to group the client population into clusters with jointly trainable data distributions. In contrast to existing FMTL approaches, CFL does not require any modifications to the FL communication protocol to be made, is applicable to general non-convex objectives (in particular deep neural networks) and comes with strong mathematical guarantees on the clustering quality. CFL is flexible enough to handle client populations that vary over time and can be implemented in a privacy preserving way. As clustering is only performed after Federated Learning has converged to a stationary point, CFL can be viewed as a post-processing method that will always achieve greater or equal performance than conventional FL by allowing clients to arrive at more specialized models. We verify our theoretical analysis in experiments with deep convolutional and recurrent neural networks on commonly used Federated Learning datasets.
Introduction: The Limits of Standard Federated Learning
Standard Federated Learning (FL) operates on a critical assumption: a single global parameter configuration can minimize risk for all participating clients simultaneously. This "one-size-fits-all" approach is highly effective when data is independent and identically distributed (IID). However, in real-world scenarios, client data distributions often diverge significantly.
When client distributions are "incongruent" — for example, when different users have conflicting labeling schemes or distinct linguistic patterns — a single model cannot satisfy everyone. In such cases, the global model converges to a suboptimal stationary point that fails to perform well for any specific group.
Clustered Federated Learning (CFL)
To overcome these limitations, we propose Clustered Federated Learning (CFL). CFL is a model-agnostic multi-task learning approach that automatically identifies and groups clients with similar data distributions without requiring access to their raw data.
Geometric Clustering Mechanism
The core innovation of CFL lies in its use of the geometric properties of the loss surface. The process follows these steps:
- Stationary Point Convergence: Standard FL is run until the global model reaches a stationary point where the average gradient across all clients is near zero.
- Conflict Detection: At this point, if clients have conflicting data, their individual gradients (or weight updates) will not be zero. Instead, they will point in opposing directions, effectively canceling each other out in the global average.
- Cosine Similarity Clustering: CFL calculates the cosine similarity between these client weight updates. A negative or low similarity indicates that clients are pulling the model in different directions.
- Recursive Bi-partitioning: If the similarity falls below a certain threshold, the server splits the client population into two clusters. This process repeats recursively, forming a "parameter tree" of increasingly specialized models.
Privacy and Flexibility
CFL maintains privacy by allowing clients to apply orthonormal transformations to their updates before transmission. These transformations preserve the angles and norms necessary for cosine similarity clustering while masking the underlying data. Furthermore, CFL is a "safe" upgrade; since clustering occurs after global convergence, one can always revert to the standard FL solution if specialization does not yield improvements.
Experimental Evaluation
We evaluated CFL across several challenging non-IID scenarios:
- MNIST with Label Swaps: By swapping labels (e.g., '3' and '5') for certain clients, we simulated incongruent distributions. CFL robustly identified these clusters even with as few as 20 data points per client.
- CIFAR-10 with Permuted Labels: In a task where standard FL stagnated at 20% accuracy, CFL's recursive splitting allowed specialized models to achieve nearly 60% accuracy.
- Ag-News Language Modeling: Using LSTMs on a news corpus, CFL reduced average test perplexity from 42 (standard FL) to below 36 by clustering clients based on news topics (e.g., Sports vs. Sci/Tech).
Key Insights
- The Geometry of Conflict: Disagreement between clients is encoded in the angles of their gradient updates at the global stationary point.
- Weight Updates vs. Gradients: Accumulated weight updates over several local epochs provide better clustering separation than raw single-batch gradients.
- Model-Agnosticism: Unlike previous methods limited to convex problems, CFL is fully compatible with deep, non-convex neural networks.
Conclusion
Clustered Federated Learning provides a robust framework for handling non-IID data in distributed environments. By exploiting the geometric conflicts in client updates, CFL allows for the automated discovery of client clusters and the training of specialized models, significantly outperforming standard Federated Learning in heterogeneous settings.
Summary updated