Abstract
Federated learning (FL) on heterogeneous data (non-IID data) has recently received great attention. Most existing methods focus on studying the convergence guarantees for the global objective. While these methods can guarantee the decrease of the global objective in each communication round, they fail to ensure risk decrease for each client. In this paper, to address the problem, we propose FedCOME, which introduces a consensus mechanism to enforce decreased risk for each client after each training round. In particular, we allow a slight adjustment to a client's gradient on the server side, which generates an acute angle between the corrected gradient and the original ones of other clients. We theoretically show that the consensus mechanism can guarantee the convergence of the global objective. To generalize the consensus mechanism to the partial participation FL scenario, we devise a novel client sampling strategy to select the most representative clients for the global data distribution. Finally, we conduct extensive experiments on four benchmark datasets to show the superiority of FedCOME against other state-of-the-art methods in terms of effectiveness, efficiency and fairness.
Introduction
In Federated Learning, a central server coordinates training across numerous edge devices (clients) without accessing their private data. A primary challenge is data heterogeneity (non-IID data), where local data distributions vary significantly between clients. Standard algorithms like FedAvg or FedProx minimize the average error across all clients. However, a critical flaw exists: while the global average might improve, the model performance can actually degrade for specific individual clients during a training round. This "client drift" slows convergence and leads to unfair models. FedCOME shifts the perspective from global objective optimization to guaranteed local risk reduction.
The FedCOME Framework
FedCOME (Federated learning via COnsensus MEchanism) addresses heterogeneity through two primary innovations:
1. The Consensus Mechanism
On the server side, instead of simple gradient averaging, FedCOME performs a constrained adjustment to each client's gradient. By solving a Quadratic Programming (QP) problem, the server ensures that the corrected gradient for any given client forms an acute angle (dot product $\ge 0$) with the original gradients of other clients. This mathematical consensus ensures that updates intended for the collective do not inadvertently harm individual local performance.
2. Representative Client Sampling
To handle partial participation, FedCOME maintains a similarity table based on historical gradient directions. Using simulated annealing, it selects a subset of clients that are the least similar to each other. By training on these diverse "representative" clients, the consensus reached is more likely to benefit the entire population, including idle clients.
Experimental Evaluation
The framework was evaluated on four standard benchmarks: MNIST, FEMNIST, CIFAR-10, and CIFAR-100, using pathological non-IID splits where clients only observe a subset of categories.
Key Results
- Higher Accuracy: FedCOME consistently outperformed state-of-the-art baselines. On the challenging CIFAR-100 dataset, FedCOME achieved 37.66% accuracy, surpassing the runner-up FedDyn (35.92%).
- Robustness to Heterogeneity: In extreme data skew scenarios (e.g., CIFAR-10 with 2 classes per client), FedCOME maintained a lead of nearly 1.5% over competitors.
- Superior Fairness: Accuracy distributions across clients were significantly sharper for FedCOME, indicating fewer clients were left behind with poor performance compared to traditional methods.
- Sampling Efficiency: The representative sampling strategy improved accuracy by 2.77% over random sampling in partial participation scenarios.
Conclusion
FedCOME demonstrates that local progress is a prerequisite for robust global success. By treating local improvement as a hard constraint through a consensus mechanism, the framework achieves faster convergence and better fairness on heterogeneous data. While the server-side QP solver introduces some overhead, the gains in efficiency and model quality make it a compelling approach for real-world federated systems.
Summary updated