Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators

Sikai Bai, Shuaicheng Li, Weiming Zhuang, Jie Zhang, Kunlin Yang, Jun Hou, Shuai Yi, Shuai Zhang, Junyu GaoAAAI 2024 · DOI

Abstract: Federated Semi-supervised Learning (FSSL) faces significant hurdles in real-world deployments due to data imbalances across clients (external) and within local datasets (internal). We present FedDure (Federated Dual Regulators), a novel framework that employs bi-level optimization to manage these imbalances. By utilizing a coarse-grained regulator for global consistency and a fine-grained regulator for instance-level weighting, FedDure achieves substantial accuracy improvements—exceeding 11% on standard benchmarks—and demonstrates remarkable robustness to data heterogeneity.

Introduction

Federated Learning (FL) has emerged as a privacy-preserving paradigm for distributed machine learning. However, the scarcity of labeled data in edge devices necessitates Federated Semi-supervised Learning (FSSL). Standard FSSL approaches often assume relatively balanced data distributions, an assumption that collapses in the "messy reality" of real-world applications. Two critical challenges persist: External Imbalance, where divergent client data distributions lead to model drift, and Internal Imbalance, where local labeled data fails to represent the unlabeled pool. FedDure is designed to navigate these challenges by treating labeled data as a supervisory signal for the entire learning process.

The FedDure Framework

The core of FedDure lies in its bi-level optimization strategy, which alternates between updating local regulators and the primary model. This approach utilizes two distinct components:

  • Coarse-grained Regulator (C-reg): This component tracks the "learning effect" by measuring how updates from unlabeled data influence performance on the labeled validation set. It serves as a reward coefficient, dampening updates that might lead the model astray and incorporating global server knowledge to mitigate client drift.
  • Fine-grained Regulator (F-reg): Moving beyond the rigid "pass/fail" thresholds of traditional pseudo-labeling, F-reg is a lightweight neural network that learns to assign "soft weights" to individual unlabeled samples. This allows the model to extract valuable information even from low-confidence samples that are statistically informative.
The FedDure architecture illustrating the interaction between the global server and the local dual-regulator system.

Experimental Evaluation

FedDure was benchmarked against CIFAR-10, Fashion-MNIST, and CINIC-10 under a rigorous (DIR, DIR) setting. This configuration uses Dirichlet distributions to simulate extreme data heterogeneity across clients and within local storage, representing the most challenging scenarios for FSSL.

Key Results and Discussion

Experimental results highlight FedDure's superiority over existing state-of-the-art (SOTA) methods like FedMatch and FixMatch-based federated models.

  • Accuracy Improvements: FedDure achieved accuracy boosts of over 11% on CIFAR-10 and CINIC-10 in highly imbalanced settings.
  • Robustness to Heterogeneity: Unlike baseline methods whose performance "falls off a cliff" as imbalance increases, FedDure remains stable, effectively bridging the gap between semi-supervised and fully supervised learning.
  • Informative Weighting: By learning instance weights, FedDure avoids the pitfalls of "confidence-based" filtering, which often ignores informative but low-probability samples in imbalanced data regimes.
Performance comparison of FedDure against SOTA FSSL methods and supervised baselines.

Conclusion and Future Directions

FedDure demonstrates that dual regulators can effectively supervise the learning process in imbalanced federated environments. While the current work focuses on image classification, future research will explore its applicability to Natural Language Processing (NLP) and time-series data, as well as further optimizing the computational efficiency of the bi-level update steps.

Summary updated

BibTeX
@inproceedings{bai2024combating,
  author = {Sikai Bai and Shuaicheng Li and Weiming Zhuang and Jie Zhang and Kunlin Yang and Jun Hou and Shuai Yi and Shuai Zhang and Junyu Gao},
  title = {{Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i10.28974},
}
Figure