Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence

Shu Zheng, Tiandi Ye, Xiang Li, Ming Gao

TL;DR

FedCOME is a novel federated learning (FL) framework that tackles data heterogeneity by enforcing a consensus mechanism. Unlike standard methods that only guarantee global improvement, FedCOME ensures that the risk for each individual client decreases in every training round, leading to faster convergence and better fairness on non-IID data.

What Problem Does This Paper Address?

In Federated Learning, a central server coordinates training across thousands of edge devices (clients) without ever seeing their private data. A massive hurdle here is data heterogeneity (non-IID data): your phone’s photo library looks nothing like mine.

Most existing FL algorithms (like FedAvg or FedProx) focus on minimizing the average error across all clients. However, the authors point out a subtle but critical flaw: while the global average might improve in a given round, the model might actually get worse for specific individual clients. This "client drift" slows down convergence and leads to unfair models that perform poorly for certain users. The authors argue that if we can guarantee every client benefits in every round, the global model will naturally converge more robustly.

Methods

The authors introduce FedCOME (Federated learning via COnsensus MEchanism), which consists of two main components:

  1. The Consensus Mechanism: On the server side, instead of just averaging gradients, the server performs a "slight adjustment" to each client's gradient. It solves a Quadratic Programming (QP) problem to ensure that the corrected gradient for one client forms an acute angle (dot product $\ge 0$) with the original gradients of other clients. This mathematical "consensus" ensures that an update intended to help the group doesn't inadvertently harm an individual client's local performance.
  2. Representative Client Sampling: In real-world FL, we can't train on every device at once. FedCOME maintains a similarity table ($S$) on the server based on past gradient directions. It uses simulated annealing to select a subset of clients that are the least similar to each other. By picking the most diverse/representative "ambassadors" for the global population, the consensus reached by the active group is more likely to benefit the idle clients as well.

Datasets

The framework was evaluated on four standard FL benchmarks:

  • MNIST: Handwritten digits (10 classes, 100 clients).
  • FEMNIST: A more complex, naturally heterogeneous version of MNIST including letters.
  • CIFAR-10 & CIFAR-100: Popular image classification datasets partitioned into "pathological" non-IID splits (where each client only sees a few categories).

Key Insights and Ideas

  • Local Progress = Global Success: The core "aha moment" is shifting the perspective from global objective optimization to local risk reduction. By treating local improvement as a hard constraint, global convergence becomes a byproduct of collective local progress.
  • Gradients as Proxies for Similarity: The paper uses the cosine similarity of gradients to build a map of how "different" clients are. This allows the server to act like a smart curator, picking a diverse "jury" of clients for each round rather than relying on blind luck (random sampling).
  • Minimal Intervention: The consensus mechanism doesn't throw away local updates; it applies the smallest possible change (in a least-squares sense) to align them, preserving as much of the original local knowledge as possible.

Key Results

FedCOME consistently outperformed state-of-the-art baselines like FedProx, SCAFFOLD, and FedDyn:

  • Higher Accuracy: On the challenging CIFAR-100 dataset with full participation, FedCOME achieved 37.66% accuracy, significantly beating the runner-up FedDyn (35.92%).
  • Robustness to Heterogeneity: When tested on CIFAR-10 with extreme data skew (only 2 classes per client), FedCOME maintained a lead of nearly 1.5% over the next best method.
  • Superior Fairness: The authors plotted the distribution of accuracy across clients. FedCOME showed a much "sharper" distribution than competitors, meaning fewer clients were left behind with poor performance.
  • Sampling Efficiency: In partial participation scenarios (only 10% of clients active), FedCOME’s sampling strategy improved accuracy by 2.77% compared to simple random sampling.

Limitations and Future Work

  • Computational Overhead: Solving a Quadratic Programming (QP) problem on the server adds computational cost. While the authors argue this is manageable since the number of selected clients ($M$) is much smaller than the total population ($N$), it could become a bottleneck for very large active sets.
  • Privacy Risks: Like most gradient-based FL methods, there is a theoretical risk of "gradient leakage" where a malicious server could try to reconstruct private data. The authors suggest this can be mitigated with standard techniques like differential privacy or gradient clipping, but they did not explicitly test these integrations.
  • Statelessness: While the paper mentions the method allows clients to be "stateless" (not needing to remember previous rounds), the server still needs to maintain the similarity table, which requires tracking client identities over time.

Summary updated

Abstract

Federated learning (FL) on heterogeneous data (non-IID data) has recently received great attention. Most existing methods focus on studying the convergence guarantees for the global objective. While these methods can guarantee the decrease of the global objective in each communication round, they fail to ensure risk decrease for each client. In this paper, to address the problem,we propose FedCOME, which introduces a consensus mechanism to enforce decreased risk for each client after each training round. In particular, we allow a slight adjustment to a client's gradient on the server side, which generates an acute angle between the corrected gradient and the original ones of other clients. We theoretically show that the consensus mechanism can guarantee the convergence of the global objective. To generalize the consensus mechanism to the partial participation FL scenario, we devise a novel client sampling strategy to select the most representative clients for the global data distribution. Training on these selected clients with the consensus mechanism could empirically lead to risk decrease for clients that are not selected. Finally, we conduct extensive experiments on four benchmark datasets to show the superiority of FedCOME against other state-of-the-art methods in terms of effectiveness, efficiency and fairness. For reproducibility, we make our source code publicly available at: \url{https://github.com/fedcome/fedcome}.
BibTeX
@inproceedings{zheng2024federated,
  author = {Shu Zheng and Tiandi Ye and Xiang Li and Ming Gao},
  title = {{Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446892},
}

Loading related content...

Figure