Introduction
Federated Learning (FL) enables collaborative model training across decentralized clients while preserving data privacy. However, real-world datasets often contain "noisy" labels due to human error or sensor issues. While deep neural networks typically learn clean patterns before memorizing noise—a phenomenon addressed by Early-Learning Regularization (ELR) in centralized settings—this challenge is amplified in FL. Data heterogeneity (non-i.i.d. distributions) across clients leads to inconsistent noise patterns, resulting in a "dual nature of memorization" where both local clients and the global server eventually adopt incorrect labels.
Federated Label-mixture Regularization (FLR)
To combat this, the authors propose Federated Label-mixture Regularization (FLR). The core mechanism involves generating robust pseudo-labels to guide the model away from noisy supervision.
FLR utilizes two primary strategies:
- Temporal Ensembling: It maintains a running average of model predictions over time, smoothing out the stochasticity of noisy labels.
- Label Mixture: Pseudo-labels are synthesized by blending predictions from the global model (providing a broad, stable perspective) and the local model (capturing client-specific data distributions).
By penalizing deviations from these blended pseudo-labels, FLR ensures the model retains the clean patterns learned during the early stages of training.
Experimental Evaluation
The framework was tested on several benchmarks including CIFAR-10/100 (synthetic noise), CIFAR-10N (human-labeled noise), and Clothing1M (web-crawled noise).
Key Results
FLR consistently outperformed standard Federated Averaging (FedAvg) and existing noisy-label methods. For instance, on CIFAR-100 with 80% noise, FLR achieved approximately 40% accuracy, a significant improvement over the ~25% seen with FedAvg. On the Clothing1M dataset, FLR reached a state-of-the-art accuracy of 70.11%.
Visual analysis via t-SNE plots confirms that FLR effectively prevents the model from collapsing different classes together, a common failure mode when memorizing noise.
Insights and Limitations
A critical finding is the necessity of the global model's perspective. In non-i.i.d. settings, relying solely on local predictions leads to catastrophic memorization. The global model acts as a stabilizer because the server-side model tends to memorize noise more slowly than individual clients.
Despite its success, FLR introduces some computational overhead due to the need to store running averages for each data point. Future work may focus on optimizing this for resource-constrained edge devices and extending the approach to handle feature-level noise or large-scale language models.
Conclusion
FLR provides a robust solution for Federated Learning in the presence of noisy labels. By leveraging the synergy between global and local knowledge through label mixture, it effectively mitigates the risks of both local and global memorization, paving the way for more reliable decentralized AI.
Summary updated