Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence

Shu Zheng, Tiandi Ye, Xiang Li, Ming Gao

Abstract

Federated learning (FL) on heterogeneous data (non-IID data) has recently received great attention. Most existing methods focus on studying the convergence guarantees for the global objective. While these methods can guarantee the decrease of the global objective in each communication round, they fail to ensure risk decrease for each client. In this paper, to address the problem, we propose FedCOME, which introduces a consensus mechanism to enforce decreased risk for each client after each training round. In particular, we allow a slight adjustment to a client's gradient on the server side, which generates an acute angle between the corrected gradient and the original ones of other clients. We theoretically show that the consensus mechanism can guarantee the convergence of the global objective. To generalize the consensus mechanism to the partial participation FL scenario, we devise a novel client sampling strategy to select the most representative clients for the global data distribution. Finally, we conduct extensive experiments on four benchmark datasets to show the superiority of FedCOME against other state-of-the-art methods in terms of effectiveness, efficiency and fairness.

Introduction

In Federated Learning, a central server coordinates training across numerous edge devices (clients) without accessing their private data. A primary challenge is data heterogeneity (non-IID data), where local data distributions vary significantly between clients. Standard algorithms like FedAvg or FedProx minimize the average error across all clients. However, a critical flaw exists: while the global average might improve, the model performance can actually degrade for specific individual clients during a training round. This "client drift" slows convergence and leads to unfair models. FedCOME shifts the perspective from global objective optimization to guaranteed local risk reduction.

The FedCOME Framework

FedCOME (Federated learning via COnsensus MEchanism) addresses heterogeneity through two primary innovations:

1. The Consensus Mechanism

On the server side, instead of simple gradient averaging, FedCOME performs a constrained adjustment to each client's gradient. By solving a Quadratic Programming (QP) problem, the server ensures that the corrected gradient for any given client forms an acute angle (dot product $\ge 0$) with the original gradients of other clients. This mathematical consensus ensures that updates intended for the collective do not inadvertently harm individual local performance.

Individual client loss curves showing consistent risk reduction across the population.

2. Representative Client Sampling

To handle partial participation, FedCOME maintains a similarity table based on historical gradient directions. Using simulated annealing, it selects a subset of clients that are the least similar to each other. By training on these diverse "representative" clients, the consensus reached is more likely to benefit the entire population, including idle clients.

Experimental Evaluation

The framework was evaluated on four standard benchmarks: MNIST, FEMNIST, CIFAR-10, and CIFAR-100, using pathological non-IID splits where clients only observe a subset of categories.

Accuracy comparison of FedCOME against state-of-the-art baselines on four benchmark datasets.

Key Results

  • Higher Accuracy: FedCOME consistently outperformed state-of-the-art baselines. On the challenging CIFAR-100 dataset, FedCOME achieved 37.66% accuracy, surpassing the runner-up FedDyn (35.92%).
  • Robustness to Heterogeneity: In extreme data skew scenarios (e.g., CIFAR-10 with 2 classes per client), FedCOME maintained a lead of nearly 1.5% over competitors.
  • Superior Fairness: Accuracy distributions across clients were significantly sharper for FedCOME, indicating fewer clients were left behind with poor performance compared to traditional methods.
  • Sampling Efficiency: The representative sampling strategy improved accuracy by 2.77% over random sampling in partial participation scenarios.

Conclusion

FedCOME demonstrates that local progress is a prerequisite for robust global success. By treating local improvement as a hard constraint through a consensus mechanism, the framework achieves faster convergence and better fairness on heterogeneous data. While the server-side QP solver introduces some overhead, the gains in efficiency and model quality make it a compelling approach for real-world federated systems.

Summary updated

Abstract

Federated learning (FL) on heterogeneous data (non-IID data) has recently received great attention. Most existing methods focus on studying the convergence guarantees for the global objective. While these methods can guarantee the decrease of the global objective in each communication round, they fail to ensure risk decrease for each client. In this paper, to address the problem,we propose FedCOME, which introduces a consensus mechanism to enforce decreased risk for each client after each training round. In particular, we allow a slight adjustment to a client's gradient on the server side, which generates an acute angle between the corrected gradient and the original ones of other clients. We theoretically show that the consensus mechanism can guarantee the convergence of the global objective. To generalize the consensus mechanism to the partial participation FL scenario, we devise a novel client sampling strategy to select the most representative clients for the global data distribution. Training on these selected clients with the consensus mechanism could empirically lead to risk decrease for clients that are not selected. Finally, we conduct extensive experiments on four benchmark datasets to show the superiority of FedCOME against other state-of-the-art methods in terms of effectiveness, efficiency and fairness. For reproducibility, we make our source code publicly available at: \url{https://github.com/fedcome/fedcome}.
BibTeX
@inproceedings{zheng2024federated,
  author = {Shu Zheng and Tiandi Ye and Xiang Li and Ming Gao},
  title = {{Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446892},
}
Figure