Abstract
Federated Learning (FL) often faces the challenge of "noise heterogeneity," where different clients possess varying levels of label noise. Traditional methods often fail by either discarding noisy clients entirely or applying uniform denoising strategies that leave significant "noise residue." We introduce FedES (Federated Early-Stopping), a novel framework designed to adaptively handle noise across a continuous spectrum. FedES employs a two-stage approach: first, it estimates local noise rates using the distribution of Parameter Criticality (CoP) and Signed Earth Mover’s Distance (EMD) without requiring access to raw data. Second, it implements Parameter-Adaptive Updating through Selective Gradient Decay (SeGD), effectively "early-stopping" parameters prone to noise memorization while allowing cleaner parameters to continue learning. Extensive evaluations on CIFAR-10, CIFAR-100, and the real-world Clothing1M dataset demonstrate that FedES significantly outperforms state-of-the-art methods, maintaining robustness even in non-IID settings.
Introduction
Federated Learning allows multiple parties to train a shared model without sharing raw data. However, real-world data is frequently mislabeled. Most existing research assumes a binary classification of clients as either "clean" or "noisy." In reality, noise exists on a spectrum—one participant might have a 10% error rate, while another has 40%. FedES addresses this by adaptively handling varying noise rates across clients without requiring a central "clean" validation set.
Methodology
The FedES framework operates in two distinct stages to manage noise heterogeneity effectively.
Stage 1: Federated Noise Estimation
To quantify noise without seeing local data, FedES analyzes the Criticality of Parameters (CoP). This metric measures a parameter's contribution to the model's output. Since neural networks typically learn clean patterns before memorizing noise, the CoP distribution differs significantly between clean and noisy clients. FedES uses a Signed Earth Mover’s Distance (EMD) to compare local CoP distributions against the global average, providing a precise estimate of data quality for each client.
Stage 2: Parameter-Adaptive Updating
Once noise levels are estimated, the system applies Selective Gradient Decay (SeGD). Rather than halting training for the entire model, noisy clients only update their most "critical" parameters. This targeted approach prevents the model from overfitting to noisy labels while still extracting useful information from the client's cleaner data subsets. Finally, Noise-Aware Global Aggregation (NaAgg) assigns higher weights to cleaner clients during the merge process.
Key Insights and Results
The "Memorization Effect" as a Diagnostic Tool
FedES exploits the observation that neural networks learn simple, clean patterns first. By identifying which parameters are learning useful features versus those starting to overfit to noise, the framework can apply regularization selectively. Gradients act as an information-rich fingerprint for data quality, allowing the server to distinguish between "different data" (Non-IID) and "bad data" (noisy).
Noise Estimation Accuracy
A critical component of FedES is its ability to accurately estimate the real quality of client data. As shown in the experimental results, the estimated quality $\rho$ closely tracks the real quality $q$ across different datasets, significantly outperforming baseline methods like S-FedAvg.
Performance Benchmarks
FedES consistently outperformed state-of-the-art (SOTA) methods across all tested scenarios:
- CIFAR-10: With high noise ($\mu=0.5$), FedES achieved 85.40% accuracy, compared to 55.59% for standard FedAvg.
- Clothing1M: On this real-world dataset, FedES reached 73.03% accuracy, the highest among all compared methods, proving its efficacy in natural, non-synthetic environments.
Conclusion
FedES moves beyond binary noise assumptions by treating data quality as a continuous variable. By combining gradient-based noise estimation with parameter-selective updates, it prevents noise memorization without sacrificing the valuable information present in noisy clients. Future work will explore the framework's robustness against adversarial poisoning and further optimize the initial "warm-up" phase required for parameter criticality estimation.
Summary updated