FedFixer: Mitigating Heterogeneous Label Noise in Federated Learning

Xinyuan Ji, Zhaowei Zhu, Wei Xi, Olga Gadyatskaya, Zilong Song, Yong Cai, Yang Liu

In the world of Federated Learning (FL), we often assume that while data is private, the labels provided by users are mostly correct. In reality, real-world data is messy. Labels are often "noisy" (wrong), and more importantly, this noise is heterogeneous—one client might consistently mistake cats for dogs, while another mistakes them for cars.

Existing methods often struggle to tell the difference between a "noisy" label (a mistake) and a "client-specific" sample (a rare but correct data point unique to one user). A new paper, FedFixer, proposes a clever dual-model architecture to solve this, ensuring that FL remains robust even when the data is riddled with inconsistent errors.

TL;DR

FedFixer introduces a dual-model "co-teaching" framework for Federated Learning to mitigate the impact of inconsistent label noise across different clients. By pairing a global model with a personalized local model and applying specific regularizers, it effectively filters out bad data without losing valuable, client-specific information.

What Problem Does This Paper Address?

Federated Learning aims to train a shared global model across many decentralized devices. However, label quality varies wildly between users. This "heterogeneous label noise" is a double-edged sword:

  1. Generalization Gap: Noisy labels degrade the global model's performance.
  2. The Identification Dilemma: Standard "small-loss" tricks (which assume samples with low loss are clean) fail in FL. A high-loss sample might be a wrong label, or it might just be a perfectly valid sample that looks different from the global average.

As shown in the paper's analysis, FL performance degrades much more sharply than Centralized Learning as the noise rate increases, highlighting a critical need for FL-specific denoising.

Figure 1: (a): The heterogeneous label noise distributions encompass diverse true class samples or varying label noise transi- tions. (b): The performance comparison between CL and FL on CIFAR-10 with the rate of label noise 0.2, 0.5, and 0.8.

Methods

The core of FedFixer is a Dual Model Structure at the client level, consisting of a global model ($w$) and an auxiliary personalized model ($\theta$).

  • Co-Teaching Strategy: During local training, the two models work in tandem. The global model selects a "clean" subset of data for the personalized model to learn from, and vice versa. This cross-pollination prevents a single model from memorizing its own mistakes.
  • Confidence Regularizer (CR): To prevent the models from becoming overconfident in noisy labels, the authors add a CR term to the loss function. This guides the model to favor predictions that align with the overall class distribution of the dataset.
  • Distance Regularizer (DR): Because local datasets are small, personalized models are prone to overfitting. The DR forces the personalized model to stay within a reasonable "distance" of the global model, acting as an anchor that prevents the local model from drifting into the weeds of noisy local data.

Datasets

The authors evaluated FedFixer on three major benchmarks:

  • MNIST: 60,000 images of handwritten digits.
  • CIFAR-10: 50,000 natural images across 10 classes.
  • Clothing1M: A massive real-world dataset containing 1 million images with inherently noisy labels from online shopping sites.

To simulate the challenges of FL, they used various "non-IID" (not identically and independently distributed) settings, varying the probability of classes appearing on different clients and the intensity of the label noise.

Key Insights and Ideas

  • The Power of Peer Review: The "aha moment" here is that the global model (which has "seen it all") and the personalized model (which knows the local context) provide different but complementary perspectives. Using one to filter data for the other breaks the feedback loop of error accumulation.
  • Regularization as an Anchor: The Distance Regularizer is crucial. Without it, the personalized model would simply memorize the local noise, eventually leading the global model astray during aggregation.
  • Stability Matters: FedFixer doesn't just perform well on average; it shows high "denoising stability." This means it consistently identifies correct labels across all clients, rather than working well for some and failing for others.

Key Results

FedFixer consistently outperformed state-of-the-art (SOTA) methods like FedCorr and RFL, particularly in high-noise scenarios:

  • High Noise Performance: On CIFAR-10 with extreme noise (where all clients have noisy labels), FedFixer achieved 62.87% accuracy, outperforming the next best method (FedCorr) by over 10%.
  • Real-World Success: On the Clothing1M dataset, FedFixer reached 70.52% accuracy, a significant jump over standard FedAvg (66.75%).
  • F-Score Distribution: The F-score (a measure of how accurately the model identifies noisy vs. clean labels) was much tighter and higher for FedFixer compared to competitors, proving its reliability across diverse client environments.

Limitations and Future Work

While FedFixer is highly effective at filtering noisy samples, it currently discards them. The authors note that label correction—actually fixing the wrong labels instead of just ignoring them—could further boost performance. Additionally, while the dual-model approach is powerful, it doubles the local memory and computation requirements for clients, which might be a constraint for very low-power IoT devices. Future research could look into "slimmer" versions of this dual-architecture.

Summary updated

Abstract

Federated Learning (FL) heavily depends on label quality for its performance. However, the label distribution among individual clients is always both noisy and heterogeneous. The high loss incurred by client-specific samples in heterogeneous label noise poses challenges for distinguishing between client-specific and noisy label samples, impacting the effectiveness of existing label noise learning approaches. To tackle this issue, we propose FedFixer, where the personalized model is introduced to cooperate with the global model to effectively select clean client-specific samples. In the dual models, updating the personalized model solely at a local level can lead to overfitting on noisy data due to limited samples, consequently affecting both the local and global models' performance. To mitigate overfitting, we address this concern from two perspectives. Firstly, we employ a confidence regularizer to alleviate the impact of unconfident predictions caused by label noise. Secondly, a distance regularizer is implemented to constrain the disparity between the personalized and global models. We validate the effectiveness of FedFixer through extensive experiments on benchmark datasets. The results demonstrate that FedFixer can perform well in filtering noisy label samples on different clients, especially in highly heterogeneous label noise scenarios.
BibTeX
@inproceedings{ji2024fedfixer,
  author = {Xinyuan Ji and Zhaowei Zhu and Wei Xi and Olga Gadyatskaya and Zilong Song and Yong Cai and Yang Liu},
  title = {{FedFixer: Mitigating Heterogeneous Label Noise in Federated Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i11.29179},
}
Figure