Semi-Federated Learning

Zhikun Chen, Daofeng Li, Ming Zhao, Sihai Zhang, Jinkang Zhu

Abstract

Federated learning (FL) enables massive distributed Information and Communication Technology (ICT) devices to learn a global consensus model without any participants revealing their own data to the central server. However, the practicality, communication expense and non-independent and identical distribution (Non-IID) data challenges in FL still need to be concerned. In this work, we propose the Semi-Federated Learning (Semi-FL) which differs from the FL in two aspects, local clients clustering and in-cluster training. A sequential training manner is designed for our in-cluster training in this paper which enables the neighboring clients to share their learning models. The proposed Semi-FL can be easily applied to future mobile communication networks and require less up-link transmission bandwidth. Numerical experiments validate the feasibility, learning performance and the robustness to Non-IID data of the proposed Semi-FL. The Semi-FL extends the existing potentials of FL.

Introduction

Standard Federated Learning (FL) allows multiple devices to train a shared model while preserving data privacy. However, it faces significant hurdles in massive IoT or 5G environments: communication bottlenecks due to simultaneous uploads, performance degradation with non-independent and identically distributed (Non-IID) data, and potential privacy leaks from frequent model updates.

To address these, we introduce Semi-Federated Learning (Semi-FL). Semi-FL shifts some of the consensus-building from the central server to local clusters of devices, introducing a hierarchical approach that reduces uplink overhead and improves robustness to data heterogeneity.

The Semi-FL Architecture

The core of Semi-FL is a two-tier architecture consisting of local client clustering and in-cluster training.

The hierarchical architecture of Semi-Federated Learning, showing clusters of clients communicating with a central server.

Local Client Clustering

Devices are grouped into clusters based on physical proximity or network topology, such as those connected to the same small cell or access point. This grouping allows for efficient local communication.

Sequential In-Cluster Training

Unlike standard FL where every device trains independently, Semi-FL employs a "relay" fashion within clusters. A model is passed sequentially: Client 1 trains on its data and passes the update to Client 2, who continues training before passing it to Client 3, and so on. Only the final "head" of the cluster uploads the aggregated model to the central server. This reduces uplink transmissions by a factor equal to the cluster size.

Experimental Setup

The framework was evaluated using the MNIST dataset. To simulate real-world challenges, two environments were created:

  • IID Setting: Each client has a uniform distribution of all 10 digits.
  • Non-IID Setting: Data is sorted by label so each client contains images of only one specific digit.

Four clustering patterns (c1-c4) were tested to analyze how intra-cluster data diversity affects performance.

Results and Discussion

Semi-FL demonstrates significant improvements in both accuracy and communication efficiency, particularly in challenging Non-IID scenarios.

Test accuracy vs. communication rounds in a Non-IID setting, comparing Semi-FL with standard FL and centralized learning.

Performance and Robustness

In Non-IID scenarios, standard FL with 100% participation reached only 88% accuracy. In contrast, Semi-FL maintained an accuracy between 89% and 98%, nearly matching Centralized Learning (CL) at ~98%. The results indicate that as long as a cluster collectively covers a diverse set of labels, the model performs exceptionally well despite individual client bias.

Efficiency and Convergence

By performing "pre-averaging" at the edge through sequential training, the updates sent to the central server are more stable. Mathematical analysis using Averaged Cosine Similarity (ACS) and Relative Euclidean Distance (RED) confirms that Semi-FL models converge closer to the ideal centralized model than standard FL.

Conclusion

Semi-FL offers a promising path for efficient distributed learning in mobile networks. By leveraging local clusters and sequential training, it drastically reduces uplink congestion and overcomes the accuracy drops typical of Non-IID data. Future work will address the inherent latency of sequential training and explore security measures against potentially malicious cluster members.

Summary updated

Abstract

Federated learning (FL) enables massive distributed Information and Communication Technology (ICT) devices to learn a global consensus model without any participants revealing their own data to the central server. However, the practicality, communication expense and non-independent and identical distribution (Non-IID) data challenges in FL still need to be concerned. In this work, we propose the Semi-Federated Learning (Semi-FL) which differs from the FL in two aspects, local clients clustering and in-cluster training. A sequential training manner is designed for our in-cluster training in this paper which enables the neighboring clients to share their learning models. The proposed Semi-FL can be easily applied to future mobile communication networks and require less up-link transmission bandwidth. Numerical experiments validate the feasibility, learning performance and the robustness to Non-IID data of the proposed Semi-FL. The Semi-FL extends the existing potentials of FL.
BibTeX
@misc{chen2020semi,
  author = {Zhikun Chen and Daofeng Li and Ming Zhao and Sihai Zhang and Jinkang Zhu},
  title = {{Semi-Federated Learning}},
  year = {2020},
  eprint = {2003.12795},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.12795},
}
Figure