Revisiting Early-Learning Regularization When Federated Learning Meets Noisy Labels

Taehyeon Kim, Donggyu Kim, Se-Young YunarXiv 2024 · DOI · arXiv

Introduction

Federated Learning (FL) enables collaborative model training across decentralized clients while preserving data privacy. However, real-world datasets often contain "noisy" labels due to human error or sensor issues. While deep neural networks typically learn clean patterns before memorizing noise—a phenomenon addressed by Early-Learning Regularization (ELR) in centralized settings—this challenge is amplified in FL. Data heterogeneity (non-i.i.d. distributions) across clients leads to inconsistent noise patterns, resulting in a "dual nature of memorization" where both local clients and the global server eventually adopt incorrect labels.

Federated Label-mixture Regularization (FLR)

To combat this, the authors propose Federated Label-mixture Regularization (FLR). The core mechanism involves generating robust pseudo-labels to guide the model away from noisy supervision.

Overview of the FLR framework, illustrating the mixture of global and local predictions to form a stable pseudo-label.

FLR utilizes two primary strategies:

  1. Temporal Ensembling: It maintains a running average of model predictions over time, smoothing out the stochasticity of noisy labels.
  2. Label Mixture: Pseudo-labels are synthesized by blending predictions from the global model (providing a broad, stable perspective) and the local model (capturing client-specific data distributions).

By penalizing deviations from these blended pseudo-labels, FLR ensures the model retains the clean patterns learned during the early stages of training.

Experimental Evaluation

The framework was tested on several benchmarks including CIFAR-10/100 (synthetic noise), CIFAR-10N (human-labeled noise), and Clothing1M (web-crawled noise).

Key Results

FLR consistently outperformed standard Federated Averaging (FedAvg) and existing noisy-label methods. For instance, on CIFAR-100 with 80% noise, FLR achieved approximately 40% accuracy, a significant improvement over the ~25% seen with FedAvg. On the Clothing1M dataset, FLR reached a state-of-the-art accuracy of 70.11%.

t-SNE visualizations comparing Cross-Entropy (CE) loss with FLR. FLR maintains distinct clusters for different classes, preventing noisy samples from "bleeding" into incorrect categories.

Visual analysis via t-SNE plots confirms that FLR effectively prevents the model from collapsing different classes together, a common failure mode when memorizing noise.

Insights and Limitations

A critical finding is the necessity of the global model's perspective. In non-i.i.d. settings, relying solely on local predictions leads to catastrophic memorization. The global model acts as a stabilizer because the server-side model tends to memorize noise more slowly than individual clients.

Despite its success, FLR introduces some computational overhead due to the need to store running averages for each data point. Future work may focus on optimizing this for resource-constrained edge devices and extending the approach to handle feature-level noise or large-scale language models.

Conclusion

FLR provides a robust solution for Federated Learning in the presence of noisy labels. By leveraging the synergy between global and local knowledge through label mixture, it effectively mitigates the risks of both local and global memorization, paving the way for more reliable decentralized AI.

Summary updated

Abstract

In the evolving landscape of federated learning (FL), addressing label noise presents unique challenges due to the decentralized and diverse nature of data collection across clients. Traditional centralized learning approaches to mitigate label noise are constrained in FL by privacy concerns and the heterogeneity of client data. This paper revisits early-learning regularization, introducing an innovative strategy, Federated Label-mixture Regularization (FLR). FLR adeptly adapts to FL's complexities by generating new pseudo labels, blending local and global model predictions. This method not only enhances the accuracy of the global model in both i.i.d. and non-i.i.d. settings but also effectively counters the memorization of noisy labels. Demonstrating compatibility with existing label noise and FL techniques, FLR paves the way for improved generalization in FL environments fraught with label inaccuracies.
BibTeX
@misc{kim2024revisiting,
  author = {Taehyeon Kim and Donggyu Kim and Se-Young Yun},
  title = {{Revisiting Early-Learning Regularization When Federated Learning Meets Noisy Labels}},
  year = {2024},
  eprint = {2402.05353},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.05353},
}
Figure