Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints

Felix Sattler, Klaus-Robert Müller, Wojciech Samek

TL;DR

Clustered Federated Learning (CFL) solves the "one-size-fits-all" problem in standard federated learning by automatically grouping clients with similar data distributions into distinct clusters. By analyzing the geometric properties of gradient updates, CFL allows for the specialized training of multiple models without requiring changes to existing communication protocols or compromising user privacy.

What Problem Does This Paper Address?

Federated Learning (FL) has become the gold standard for training machine learning models on decentralized data (like photos on smartphones or logs on IoT devices) without actually moving that data to a central server. However, standard FL makes a massive assumption: that a single global model can satisfy every client.

In the real world, data is often "incongruent." For example, if you are training a next-word predictor, a teenager’s texting style is vastly different from a retiree’s. If a single model tries to learn both simultaneously, it often ends up being mediocre for everyone. Existing solutions for "Multitask Federated Learning" often require knowing the number of groups in advance, only work for simple linear models, or require invasive changes to how the server and clients talk to each other.

Methods

The authors introduce Clustered Federated Learning (CFL), a framework that treats clustering as a recursive post-processing step.

  1. Stationary Point Convergence: The system starts by training a single global model using standard FL (like FedAvg) until it reaches a stationary point (where the global gradient is near zero).
  2. Cosine Similarity-Based Bipartitioning: The authors prove a "Separation Theorem": if clients have diverging data distributions, their individual gradients at that stationary point will point in significantly different directions. By calculating the cosine similarity between client updates, the server can identify these disagreements.
  3. Recursive Splitting: If the server detects that clients are "unhappy" with the global model (i.e., their local gradients are still large even though the global average is zero), it splits the client population into two clusters.
  4. Model Specialization: Each cluster then continues training its own specialized model. This process repeats recursively, forming a "parameter tree" of models ranging from general to highly specific.

Crucially, this method is model-agnostic, meaning it works just as well for deep neural networks as it does for simple regressions.

Datasets

The researchers evaluated CFL using three primary setups:

  • MNIST: The classic handwritten digit dataset, modified with "label-swap augmentation" (e.g., one group of clients sees '1' as '7' and vice versa) to simulate conflicting data distributions.
  • CIFAR-10: A more complex image dataset, also using label-swapping to create four distinct client clusters.
  • Ag-News: A large-scale news article corpus used for language modeling. Clients were grouped by topic (World, Sports, Business, Sci/Tech) to see if CFL could cluster them based on the semantic content of their text.

Key Insights and Ideas

  • The Geometry of Disagreement: The most elegant insight is that at a global stationary point, the sum of client gradients is zero. If the clients are incongruent, their gradients must be non-zero and pointing in opposite directions to cancel each other out. This geometric "tension" is the signal used for clustering.
  • Updates > Gradients: While the theory is based on gradients, the authors found that using weight updates (the difference between the model before and after local training) actually provides a cleaner signal for clustering in practice.
  • Privacy via Invariance: Cosine similarity is invariant to orthonormal transformations (like shuffling the order of weights). This allows clients to "scramble" their updates before sending them to the server, adding a layer of privacy that prevents the server from seeing raw weights while still allowing it to calculate similarities.
  • No A Priori Knowledge: Unlike many clustering algorithms (like K-Means), CFL does not need to be told how many clusters exist. It discovers the natural structure of the population automatically.

Key Results

  • Massive Accuracy Boosts: On the CIFAR-10 label-swap task, standard FL stagnated at roughly 20% accuracy. CFL successfully identified the four clusters and boosted validation accuracy to nearly 60%.
  • Better Language Models: On the Ag-News dataset, CFL reduced the "perplexity" (a measure of how surprised a model is by new text; lower is better) from 42 to under 36, outperforming the single-model approach.
  • Data Efficiency: The clustering mechanism is remarkably robust; on MNIST, it could correctly separate clients even if they only held 20 data samples each.

Limitations and Future Work

The authors acknowledge a few areas for further exploration:

  • Privacy Leaks: While the raw data is safe, the fact that the server knows which clients are similar to each other is itself a form of information leakage.
  • Computational Overhead: While the clustering calculation is efficient ($O(M^3)$ where $M$ is the number of clients), maintaining a tree of different models for a massive population could become a storage challenge for the server.
  • Future Directions: The authors suggest combining CFL with Differential Privacy and exploring Explainable AI techniques to understand exactly why the algorithm decided to put certain users in the same cluster.

Summary updated

Abstract

Federated learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit its popularity, it has been observed that FL yields suboptimal results if the local clients' data distributions diverge. To address this issue, we present clustered FL (CFL), a novel federated multitask learning (FMTL) framework, which exploits geometric properties of the FL loss surface to group the client population into clusters with jointly trainable data distributions. In contrast to existing FMTL approaches, CFL does not require any modifications to the FL communication protocol to be made, is applicable to general nonconvex objectives (in particular, deep neural networks), does not require the number of clusters to be known a priori, and comes with strong mathematical guarantees on the clustering quality. CFL is flexible enough to handle client populations that vary over time and can be implemented in a privacy-preserving way. As clustering is only performed after FL has converged to a stationary point, CFL can be viewed as a postprocessing method that will always achieve greater or equal performance than conventional FL by allowing clients to arrive at more specialized models. We verify our theoretical analysis in experiments with deep convolutional and recurrent neural networks on commonly used FL data sets.
BibTeX
@article{sattler2021clustered,
  author = {Felix Sattler and Klaus-Robert Müller and Wojciech Samek},
  title = {{Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2021},
  url = {https://openreview.net/forum?id=QwNmwrru8N2},
}

Loading related content...

Figure