Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints

Felix Sattler, Klaus‐Robert Müller, Wojciech Samek

Abstract

Federated learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit its popularity, it has been observed that FL yields suboptimal results if the local clients' data distributions diverge. To address this issue, we present clustered FL (CFL), a novel federated multitask learning (FMTL) framework, which exploits geometric properties of the FL loss surface to group the client population into clusters with jointly trainable data distributions. In contrast to existing FMTL approaches, CFL does not require any modifications to the FL communication protocol to be made, is applicable to general nonconvex objectives (in particular, deep neural networks), does not require the number of clusters to be known a priori, and comes with strong mathematical guarantees on the clustering quality. CFL is flexible enough to handle client populations that vary over time and can be implemented in a privacy-preserving way. As clustering is only performed after FL has converged to a stationary point, CFL can be viewed as a postprocessing method that will always achieve greater or equal performance than conventional FL by allowing clients to arrive at more specialized models. We verify our theoretical analysis in experiments with deep convolutional and recurrent neural networks on commonly used FL data sets.

Introduction: The Challenge of Non-IID Data

Federated Learning (FL) has emerged as a powerful paradigm for training models on decentralized data without compromising user privacy. However, a fundamental limitation of standard FL is the "one-size-fits-all" assumption. In many real-world scenarios, client data distributions are non-IID (not identically and independently distributed). For instance, a language model trained on diverse demographics may fail to capture the specific nuances of different user groups, leading to mediocre performance for everyone.

Federated Learning assumptions vs. failure modes when client data distributions diverge.

As illustrated in Figure 1, when data distributions are incongruent, a single global model cannot effectively separate or represent the underlying clusters. Existing multitask learning solutions often require prior knowledge of the number of clusters or invasive changes to communication protocols.

Methods: Clustered Federated Learning (CFL)

To overcome these hurdles, we propose Clustered Federated Learning (CFL). CFL treats clustering as a recursive post-processing step that leverages the geometric properties of the loss surface.

The Clustered Federated Learning (CFL) workflow: from global convergence to recursive cluster splitting.

The CFL framework, detailed in Figure 6, operates in several stages:

  1. Stationary Point Convergence: The process begins by training a global model using standard FL (e.g., FedAvg) until it reaches a stationary point.
  2. Cosine Similarity-Based Bipartitioning: At this point, if clients have divergent distributions, their individual gradients will point in opposing directions. By calculating the cosine similarity of these updates, the server can detect "disagreement" among clients.
  3. Recursive Splitting: If the server identifies significant incongruence, it splits the population into two clusters.
  4. Model Specialization: Each cluster then continues training its own specialized model. This process repeats, creating a "parameter tree" that evolves from general to specific models.

Key Insights

  • Geometric Tension: The core innovation lies in the "Separation Theorem." At a global stationary point, the sum of client gradients is zero. If clients are incongruent, their gradients must be non-zero and pointing in opposite directions to cancel out. This tension provides the signal for clustering.
  • Privacy-Preserving Design: Cosine similarity is invariant to certain transformations. This allows clients to apply privacy-preserving techniques, like weight shuffling, before sending updates, ensuring the server never sees raw weights.
  • Model-Agnosticism: Unlike previous methods limited to linear models, CFL is applicable to complex, non-convex objectives, including deep neural networks.

Experimental Results

We evaluated CFL across several benchmarks:

  • MNIST & CIFAR-10: Using label-swap augmentations to simulate conflicting distributions, CFL significantly outperformed standard FL. On CIFAR-10, accuracy jumped from a stagnant 20% to nearly 60%.
  • Ag-News: In a large-scale language modeling task, CFL successfully clustered clients by topic (e.g., Sports vs. Business), reducing model perplexity from 42 to under 36.
  • Robustness: The mechanism proved highly data-efficient, correctly separating MNIST clients with as few as 20 samples each.

Conclusion and Future Work

CFL provides a robust, privacy-preserving way to handle data heterogeneity in federated environments without requiring a priori knowledge of cluster structures. While it offers significant accuracy gains, future work will focus on mitigating potential privacy leaks regarding client similarity and optimizing the storage overhead of maintaining multiple specialized models.

Summary updated

Abstract

Federated learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit its popularity, it has been observed that FL yields suboptimal results if the local clients' data distributions diverge. To address this issue, we present clustered FL (CFL), a novel federated multitask learning (FMTL) framework, which exploits geometric properties of the FL loss surface to group the client population into clusters with jointly trainable data distributions. In contrast to existing FMTL approaches, CFL does not require any modifications to the FL communication protocol to be made, is applicable to general nonconvex objectives (in particular, deep neural networks), does not require the number of clusters to be known a priori, and comes with strong mathematical guarantees on the clustering quality. CFL is flexible enough to handle client populations that vary over time and can be implemented in a privacy-preserving way. As clustering is only performed after FL has converged to a stationary point, CFL can be viewed as a postprocessing method that will always achieve greater or equal performance than conventional FL by allowing clients to arrive at more specialized models. We verify our theoretical analysis in experiments with deep convolutional and recurrent neural networks on commonly used FL data sets.
BibTeX
@article{sattler2021clustered,
  author = {Felix Sattler and Klaus‐Robert Müller and Wojciech Samek},
  title = {{Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2021},
  url = {https://openreview.net/forum?id=QwNmwrru8N2},
}
Figure