FedES: Federated Early-Stopping for Hindering Memorizing Heterogeneous Label Noise

Bixiao Zeng, Xiaodong Yang, Yiqiang Chen, Zhiqi Shen

Abstract

Federated Learning (FL) often faces the challenge of "noise heterogeneity," where different clients possess varying levels of label noise. Traditional methods often fail by either discarding noisy clients entirely or applying uniform denoising strategies that leave significant "noise residue." We introduce FedES (Federated Early-Stopping), a novel framework designed to adaptively handle noise across a continuous spectrum. FedES employs a two-stage approach: first, it estimates local noise rates using the distribution of Parameter Criticality (CoP) and Signed Earth Mover’s Distance (EMD) without requiring access to raw data. Second, it implements Parameter-Adaptive Updating through Selective Gradient Decay (SeGD), effectively "early-stopping" parameters prone to noise memorization while allowing cleaner parameters to continue learning. Extensive evaluations on CIFAR-10, CIFAR-100, and the real-world Clothing1M dataset demonstrate that FedES significantly outperforms state-of-the-art methods, maintaining robustness even in non-IID settings.

Introduction

Federated Learning allows multiple parties to train a shared model without sharing raw data. However, real-world data is frequently mislabeled. Most existing research assumes a binary classification of clients as either "clean" or "noisy." In reality, noise exists on a spectrum—one participant might have a 10% error rate, while another has 40%. FedES addresses this by adaptively handling varying noise rates across clients without requiring a central "clean" validation set.

Methodology

The FedES framework operates in two distinct stages to manage noise heterogeneity effectively.

Stage 1: Federated Noise Estimation

To quantify noise without seeing local data, FedES analyzes the Criticality of Parameters (CoP). This metric measures a parameter's contribution to the model's output. Since neural networks typically learn clean patterns before memorizing noise, the CoP distribution differs significantly between clean and noisy clients. FedES uses a Signed Earth Mover’s Distance (EMD) to compare local CoP distributions against the global average, providing a precise estimate of data quality for each client.

Stage 2: Parameter-Adaptive Updating

Once noise levels are estimated, the system applies Selective Gradient Decay (SeGD). Rather than halting training for the entire model, noisy clients only update their most "critical" parameters. This targeted approach prevents the model from overfitting to noisy labels while still extracting useful information from the client's cleaner data subsets. Finally, Noise-Aware Global Aggregation (NaAgg) assigns higher weights to cleaner clients during the merge process.

The two-stage FedES framework: Federated Noise Estimation and Parameter-adaptive Updating.

Key Insights and Results

The "Memorization Effect" as a Diagnostic Tool

FedES exploits the observation that neural networks learn simple, clean patterns first. By identifying which parameters are learning useful features versus those starting to overfit to noise, the framework can apply regularization selectively. Gradients act as an information-rich fingerprint for data quality, allowing the server to distinguish between "different data" (Non-IID) and "bad data" (noisy).

Noise Estimation Accuracy

A critical component of FedES is its ability to accurately estimate the real quality of client data. As shown in the experimental results, the estimated quality $\rho$ closely tracks the real quality $q$ across different datasets, significantly outperforming baseline methods like S-FedAvg.

Comparison of estimated quality vs. real quality on CIFAR-10 and CIFAR-100 benchmarks.

Performance Benchmarks

FedES consistently outperformed state-of-the-art (SOTA) methods across all tested scenarios:

  • CIFAR-10: With high noise ($\mu=0.5$), FedES achieved 85.40% accuracy, compared to 55.59% for standard FedAvg.
  • Clothing1M: On this real-world dataset, FedES reached 73.03% accuracy, the highest among all compared methods, proving its efficacy in natural, non-synthetic environments.

Conclusion

FedES moves beyond binary noise assumptions by treating data quality as a continuous variable. By combining gradient-based noise estimation with parameter-selective updates, it prevents noise memorization without sacrificing the valuable information present in noisy clients. Future work will explore the framework's robustness against adversarial poisoning and further optimize the initial "warm-up" phase required for parameter criticality estimation.

Summary updated

BibTeX
@inproceedings{zeng2024fedes,
  author = {Bixiao Zeng and Xiaodong Yang and Yiqiang Chen and Zhiqi Shen},
  title = {{FedES: Federated Early-Stopping for Hindering Memorizing Heterogeneous Label Noise}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=ptucbwZ1xA},
}
Figure