RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data

Guang Yang, Mu Ke, Chunhe Song, Zhijia Yang, Tierui Gong

Abstract

Federated learning is a widely used distributed deep learning framework that protects the privacy of each client by exchanging model parameters rather than raw data. However, federated learning suffers from high communication costs, as a considerable number of model parameters need to be transmitted many times during the training process, making the approach inefficient, especially when the communication network bandwidth is limited. This article proposes RingFed, a novel framework to reduce communication overhead during the training process of federated learning. Rather than transmitting parameters between the center server and each client, as in original federated learning, in the proposed RingFed, the updated parameters are transmitted between each client in turn, and only the final result is transmitted to the central server, thereby reducing the communication overhead substantially. After several local updates, clients first send their parameters to another proximal client, not to the center server directly, to preaggregate. Experiments on two different public datasets show that RingFed has fast convergence, high model accuracy, and low communication cost.

Introduction

Federated Learning (FL) enables multiple decentralized devices to collaboratively train a shared model while keeping data localized. Despite its privacy advantages, FL faces two primary challenges:

  1. Communication Overhead: Frequent transmission of high-dimensional model parameters between clients and a central server consumes significant bandwidth.
  2. Statistical Heterogeneity (Non-IID Data): Real-world data is often non-identically and independently distributed across clients, leading to local model drift and poor global convergence.

Standard algorithms like FedAvg rely on a star topology, which often becomes a bottleneck. RingFed addresses these by introducing a ring-based communication structure for pre-aggregation.

The RingFed Framework

RingFed shifts the communication paradigm from a server-centric model to a peer-to-peer pre-aggregation strategy.

Comparison of the traditional FedAvg star topology (a) and the proposed RingFed ring topology (b).

In RingFed, clients are organized in a logical ring. Instead of immediate upload to the server, a client $k$ performs local updates and then exchanges parameters with a neighbor. A weighted exchange factor ($\gamma$) is used to blend local weights with received weights: $w_{k+1} = \gamma w_k + (1-\gamma)w_k$. This "neighborhood regularization" helps mitigate local drift by exposing clients to features from their neighbors' data distributions before the global synchronization.

Experimental Setup and Data Heterogeneity

The effectiveness of RingFed was evaluated using MNIST, Fashion-MNIST, EMNIST, and CIFAR-10. To simulate realistic statistical heterogeneity, the researchers used a Dirichlet distribution to partition data among clients.

Visualization of label distributions across 30 clients for different Dirichlet concentration parameters ($\alpha$). Lower $\alpha$ values represent higher non-IID levels.

By varying $\alpha$, the study demonstrates how RingFed handles extreme data bias, where some clients may only possess a few classes of data.

Key Results and Insights

  • Communication Efficiency: RingFed significantly reduces the volume of data transmitted to the central server. In experimental scenarios, it achieved an 84% reduction in communication costs compared to FedAvg.
  • Improved Convergence: On non-IID MNIST data, RingFed reached target accuracy in 7 rounds, whereas FedAvg required 43 rounds. The pre-aggregation step provides a smoother optimization path.
  • Accuracy Gains: RingFed consistently outperformed baselines like FedProx and SCAFFOLD. For instance, on CIFAR-10, it achieved ~73.8% accuracy, surpassing FedAvg's ~71.4%.

Conclusion

RingFed offers a robust solution to the communication and heterogeneity challenges in federated learning. By leveraging local peer-to-peer exchanges, it reduces the burden on the central server and stabilizes training on non-IID data. Future work will focus on addressing network latency from sequential dependencies and evaluating the framework on larger-scale architectures like Transformers.

Summary updated

Abstract

Federated learning is a widely used distributed deep learning framework that protects the privacy of each client by exchanging model parameters rather than raw data. However, federated learning suffers from high communication costs, as a considerable number of model parameters need to be transmitted many times during the training process, making the approach inefficient, especially when the communication network bandwidth is limited. This article proposes RingFed, a novel framework to reduce communication overhead during the training process of federated learning. Rather than transmitting parameters between the center server and each client, as in original federated learning, in the proposed RingFed, the updated parameters are transmitted between each client in turn, and only the final result is transmitted to the central server, thereby reducing the communication overhead substantially. After several local updates, clients first send their parameters to another proximal client, not to the center server directly, to preaggregate. Experiments on two different public datasets show that RingFed has fast convergence, high model accuracy, and low communication cost.
BibTeX
@misc{yang2021ringfed,
  author = {Guang Yang and Mu Ke and Chunhe Song and Zhijia Yang and Tierui Gong},
  title = {{RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data}},
  year = {2021},
  eprint = {2107.08873},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.08873},
}
Figure