FedFixer: Mitigating Heterogeneous Label Noise in Federated Learning

Xinyuan Ji, Zhaowei Zhu, Wei Xi, Olga Gadyatskaya, Zilong Song, Yong Cai, Yang Liu

TL;DR

This paper introduces FedFixer, a novel dual-model approach for Federated Learning (FL) that tackles the challenging problem of heterogeneous label noise across clients. By employing a global and a personalized model that learn from each other, FedFixer effectively identifies and filters noisy samples, significantly improving performance in complex, real-world FL scenarios.

What Problem Does This Paper Address?

Federated Learning (FL) enables collaborative model training across decentralized devices without sharing raw data, making it ideal for privacy-sensitive applications. However, FL's performance is heavily reliant on the quality of data labels. A major challenge is "heterogeneous label noise," where not only are labels incorrect, but the distribution of these noisy labels varies significantly from one client to another. This heterogeneity makes it difficult for standard FL algorithms to distinguish between genuinely noisy labels and unique, client-specific data patterns. Existing methods often struggle with this, leading to degraded model performance. FedFixer aims to solve this by developing a robust FL framework that can effectively handle these diverse and noisy label distributions.

Methods

FedFixer introduces a novel "dual model structure" at each client, consisting of a global model (shared across clients) and a personalized model (client-specific). These two models are trained collaboratively:

  • Alternating Updates: The global and personalized models are updated alternately. The personalized model helps the global model by identifying clean client-specific samples, while the global model provides a more stable learning signal for the personalized model.
  • Sample Selection: A key component is a sample selector that uses the difference between a sample's loss and the average loss to identify potentially noisy samples. This selector is guided by the peer model's predictions.
  • Regularization: To combat overfitting, especially in the personalized model which has limited local data, FedFixer employs two regularization techniques:
  • Confidence Regularizer (CR): This modifies the standard Cross-Entropy loss to penalize unconfident predictions, guiding the model to learn cleaner patterns.
  • Distance Regularizer (DR): This constrains the divergence between the personalized and global models, preventing the personalized model from drifting too far due to local noisy data.

The overall training process involves clients downloading the global model, performing local dual-model updates, and then uploading their updated global model parameters for server-side aggregation.

Datasets

The paper evaluates FedFixer on three standard benchmark datasets:

  • MNIST: A dataset of handwritten digits.
  • CIFAR-10: A dataset of 10 common object classes.
  • Clothing1M: A large-scale dataset of clothing images with noisy labels.

The experiments simulate both IID (Independent and Identically Distributed) and non-IID data distributions across clients, with varying degrees of label noise.

Key Insights and Ideas

  • The Power of Dual Models: The core idea that a personalized model can help filter noisy labels for a global model (and vice-versa) is powerful. This mutual learning process is crucial for overcoming the limitations of single-model approaches in noisy FL.
  • Heterogeneity is the Real Challenge: The paper strongly emphasizes that it's not just the presence of noise, but its heterogeneous distribution across clients that makes FL so difficult. FedFixer's design directly addresses this by allowing client-specific adaptation.
  • Overfitting is a Major Hurdle: The limited data on individual clients makes the personalized model highly susceptible to overfitting on noisy labels. The regularization techniques are essential for mitigating this and ensuring stable learning.
  • Regularization is Key to Stability: The confidence and distance regularizers are not just minor tweaks; they are critical for FedFixer's success, preventing the dual models from collapsing or diverging.

Key Results

FedFixer demonstrates strong performance across various FL scenarios with heterogeneous label noise:

  • Superior Performance in High Noise: In highly heterogeneous label noise scenarios (e.g., IID with p=1, τ=0.5), FedFixer outperforms existing state-of-the-art methods by up to 10% in accuracy.
  • Robustness: It consistently achieves excellent accuracy in both IID and non-IID settings, even with significant label noise.
  • Denoising Stability: Figure 3 shows that FedFixer exhibits a more concentrated and higher F-score distribution across clients compared to other methods, indicating stable and effective denoising.
  • Ablation Study: An ablation study confirms that each component of FedFixer (dual models, CR, DR, alternate updates) contributes to its overall performance improvement.

Limitations and Future Work

  • Computational Overhead: While effective, the dual-model structure and alternating updates likely introduce higher computational overhead compared to simpler FL algorithms.
  • Hyperparameter Tuning: The effectiveness of the regularizers (β, λ, ζ, η) might require careful tuning for different datasets and noise characteristics.
  • Future Work: The paper suggests that further exploration into more advanced sample selection strategies and adaptive regularization techniques could lead to even greater improvements. Investigating FedFixer's performance on more complex real-world datasets with diverse noise patterns would also be valuable.

Summary updated

Abstract

Federated Learning (FL) heavily depends on label quality for its performance. However, the label distribution among individual clients is always both noisy and heterogeneous. The high loss incurred by client-specific samples in heterogeneous label noise poses challenges for distinguishing between client-specific and noisy label samples, impacting the effectiveness of existing label noise learning approaches. To tackle this issue, we propose FedFixer, where the personalized model is introduced to cooperate with the global model to effectively select clean client-specific samples. In the dual models, updating the personalized model solely at a local level can lead to overfitting on noisy data due to limited samples, consequently affecting both the local and global models’ performance. To mitigate overfitting, we address this concern from two perspectives. Firstly, we employ a confidence regularizer to alleviate the impact of unconfident predictions caused by label noise. Secondly, a distance regularizer is implemented to constrain the disparity between the personalized and global models. We validate the effectiveness of FedFixer through extensive experiments on benchmark datasets. The results demonstrate that FedFixer can perform well in filtering noisy label samples on different clients, especially in highly heterogeneous label noise scenarios.
BibTeX
@inproceedings{ji2024fedfixer,
  author = {Xinyuan Ji and Zhaowei Zhu and Wei Xi and Olga Gadyatskaya and Zilong Song and Yong Cai and Yang Liu},
  title = {{FedFixer: Mitigating Heterogeneous Label Noise in Federated Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i11.29179},
}
Figure