DFCA: Decentralized Federated Clustering Algorithm

Jonas Kirch, Sebastian Becker, Tiago Koketsu Rodrigues, Stefan HarmelingSubmission to ICLR 2026 · Withdrawn

TL;DR

The paper introduces DFCA (Decentralized Federated Clustering Algorithm), a serverless, peer-to-peer federated learning framework designed to handle non-IID data across client clusters. By combining loss-based local cluster assignment with an online, sequential running-average aggregation scheme, DFCA eliminates the central server bottleneck while matching the accuracy of centralized clustered baselines within ~1%.

What Problem Does This Paper Address?

In federated learning (FL), clients collaboratively train models without centralizing their raw data. However, standard FL architectures face two distinct roadblocks:

  1. Centralized Bottlenecks: Centralized FL relies on a coordinating parameter server to average updates, which introduces communication bottlenecks, high latency, and a single point of failure.
  2. Data Heterogeneity: Client data distributions are rarely independent and identically distributed (non-IID). Clustered FL addresses this by grouping clients with similar distributions to train specialized cluster models.

While algorithms like the Iterative Federated Clustering Algorithm (IFCA) successfully cluster clients, they still rely on a central server. Existing decentralized FL alternatives often struggle in heterogeneous settings or assume soft clustering (where clients belong to multiple clusters at once). DFCA solves the challenge of performing hard clustered FL in fully decentralized, sparsely connected peer-to-peer networks.

Methods

In DFCA, each client stores $k$ models (one for each cluster) and is connected to neighbors via an undirected communication graph. Training proceeds over rounds with three core steps:

  1. Cluster Assignment: Each client evaluates its local loss across all $k$ current models and assigns itself to the cluster that yields the lowest empirical loss: $$c(i) \leftarrow \arg\min_j F_{\text{client}}(\theta_{i,j}, D_i)$$
  2. Local Update: The client updates only the model parameters $\theta_{i, c(i)}$ belonging to its assigned cluster using local stochastic gradient descent (SGD).
  3. Decentralized Aggregation: Clients exchange models with immediate graph neighbors:
  4. Clients only broadcast the model they personally updated ($\theta_{i, c(i)}$).
  5. Rather than waiting for a synchronous batch of updates, clients update their models using an online sequential running average: $$\theta_{i,j} \leftarrow \frac{r}{r+1}\theta_{i,j} + \frac{1}{r+1}\theta_{m,j}$$ where $r$ tracks incoming neighbor updates for cluster $j$. This eliminates the need to buffer all neighbor updates in memory and naturally accommodates asynchronous arrival.

The authors evaluate two initialization methods:

  • DFCA-GI (Global Initialization): All clients initialize from identical seeds or a pre-broadcast model.
  • DFCA-LI (Local Initialization): Clients initialize all $k$ models locally and independently.

The authors prove that after a finite burn-in period $\tau$, cluster assignments stabilize to the true clusters, reducing DFCA to $k$ independent decentralized SGD instances with standard convergence rates under PL or convexity assumptions.

Datasets

The authors evaluated DFCA on four image classification benchmarks partitioned across clients with artificial rotation (0°, 90°, 180°, or 270°) to simulate distinct cluster distributions:

  • MNIST: $N=240$ (or 200) clients, $k=4$ rotations, using a Multilayer Perceptron (MLP).
  • EMNIST (Balanced): $N=100$ ($k=2$) and $N=200$ ($k=4$) clients, using a 2-layer Convolutional Neural Network (CNN).
  • CIFAR-10: $N=50$ clients, $k=2$ rotations, using a CNN.
  • FEMNIST: $N=400$ clients partitioned by writer, $k=4$ rotations.

No new datasets were introduced; the authors applied standard rotation augmentations commonly used in clustered FL literature.

Key Insights and Ideas

  • Cluster Training vs. Network-Wide Mixing: A client only executes gradient descent on its assigned cluster model, but it continues to average neighbor updates for all $k$ cluster models. This allows all models to circulate through the graph, ensuring every node eventually possesses all $k$ trained cluster models.
  • Sequential Running Average as Gossip: Replacing synchronous batch averaging with a step-by-step running average preserves the expected consensus properties of doubly-stochastic gossip while lowering peak memory requirements and mitigating straggler delays.
  • Closing the DFL–CFL Gap: While decentralized learning often experiences a performance drop compared to centralized architectures, DFCA consistently stays within ~1% test accuracy of centralized IFCA.
  • Connectivity Threshold: Graph connectivity as low as $p = 0.15$ (Erdős–Rényi) is sufficient to achieve optimal performance; denser graphs yield negligible performance gains.

Key Results

  • Superiority over Decentralized Baselines: DFCA consistently outperformed decentralized baselines FedSPD (soft clustering) and DFedAvgM across all benchmarks:
  • MNIST ($k=2$): DFCA-GI: 93.7%, DFCA-LI: 92.9% vs. FedSPD: 86.2%, DFedAvgM: 91.4%.
  • EMNIST ($k=2$): DFCA-GI: 85.6%, DFCA-LI: 85.3% vs. FedSPD: 79.7%, DFedAvgM: 73.5%.
  • CIFAR-10 ($k=2$): DFCA-GI: 81.5%, DFCA-LI: 80.4% vs. FedSPD: 78.9%, DFedAvgM: 76.0%.
  • Parity with Centralized IFCA: Across both 2-cluster and 4-cluster scenarios (e.g., EMNIST $k=4$: DFCA-GI 85.3% vs. IFCA 84.4%; FEMNIST $k=4$: DFCA-GI 87.1% vs. IFCA 88.2%), DFCA performed within ~1% of centralized IFCA.
  • Initialization Impact: DFCA-GI converges faster than DFCA-LI, but DFCA-LI reaches nearly identical final accuracy once clustering assignments stabilize.

Limitations and Future Work

  • Hard Clustering Assumption: DFCA assumes each client holds data from exactly one distribution/cluster, which does not cover scenarios where clients hold mixtures of multiple distributions (soft clustering).
  • Asynchronous Evaluation: While sequential averaging theoretically supports asynchronous messaging, empirical experiments simulated synchronous communication rounds over static Erdős–Rényi graphs. The authors note that analyzing true asynchronous updates in practical deployments remains future work.
  • Scale: Experiments were conducted on standard image classification benchmarks with modest network sizes ($N \le 400$); evaluation on larger real-world datasets is left for future exploration.

Summary updated

Abstract

Clustered Federated Learning has emerged as an effective approach for handling heterogeneous data across clients by partitioning them into clusters with similar or identical data distributions. However, most existing methods, including the Iterative Federated Clustering Algorithm (IFCA), rely on a central server to coordinate model updates which creates a bottleneck and single point of failure, limiting their applicability in more realistic decentralized learning settings. In this work, we introduce DFCA, a fully decentralized clustered Federated Learning algorithm that enables clients to collaboratively train cluster-specific models without central coordination. DFCA uses a sequential running average to aggregate models from neighbors as updates arrive, providing a communication-efficient alternative to batch aggregation while maintaining clustering performance. Our experiments on various datasets demonstrate that DFCA outperforms other decentralized algorithms and performs comparably to centralized IFCA, even under sparse connectivity, highlighting its robustness and practicality for dynamic real-world decentralized networks.
BibTeX
@misc{kirch2025dfca,
  author = {Jonas Kirch and Sebastian Becker and Tiago Koketsu Rodrigues and Stefan Harmeling},
  title = {{DFCA: Decentralized Federated Clustering Algorithm}},
  year = {2025},
  eprint = {2510.15300},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.15300},
}
Figure