How to Prevent the Continuous Damage of Noises to Model Training?

Xiaotian Yu, Yang Jiang, Tianqi Shi, Zunlei Feng, Yuexuan Wang, Mingli Song, Li SunCVPR 2023 · DOI

Abstract

Training deep neural networks on datasets with noisy labels often results in "continuous damage," where incorrect gradients systematically derail model optimization. This paper proposes the Gradient Switching Strategy (GSS), a pluggable method that replaces fixed, erroneous gradient directions with a probabilistic "direction pool." By neutralizing noise through exploration and canceling systematic bias, GSS significantly improves robustness. Evaluated on benchmarks like CIFAR and Clothing1M, GSS reduces the performance gap to clean-label limits and achieves state-of-the-art results.

Introduction: The Challenge of Noisy Labels

In real-world scenarios, large-scale datasets are frequently plagued by noisy labels—instances where data is incorrectly tagged (e.g., a dog labeled as a cat). Standard deep learning frameworks assume labels are ground truth, leading models to learn incorrect patterns. Existing solutions typically rely on robust loss functions or sample screening. However, these methods share a critical flaw: they depend on the model's own discriminative power to identify noise. Early in training, an imperfect model inevitably misses many mislabeled samples. These missed samples provide a consistent, misleading gradient direction that accumulates over time, creating a significant performance gap compared to clean-data training.

The Gradient Switching Strategy (GSS)

The core innovation of this work is the Gradient Switching Strategy (GSS). Rather than following a single gradient direction dictated by a potentially incorrect label, GSS maintains a gradient direction pool for every sample in the dataset.

The GSS mechanism maintains a direction pool that updates over time, allowing uncertain samples to explore different directions to neutralize noise.

Mechanism Overview

  1. The Direction Pool: For each sample, the pool contains potential gradient directions for all possible classes, each assigned a specific probability.
  2. Probabilistic Switching: During training iterations, the model randomly selects a direction from the pool based on these probabilities rather than relying solely on the annotated label.
  3. Iterative Updates: The pool is updated every epoch using a weighted combination of original labels (for early convergence), predicted labels (leveraging model intelligence), and random labels (to force exploration in uncertain samples).

This "randomness as a shield" ensures that for samples where the model is uncertain, the gradient jumps between directions. This exploration prevents the model from overfitting to specific noise by neutralizing the misleading pull of a fixed, wrong label.

Experimental Evaluation

The authors evaluated GSS across several benchmarks, including synthetic noise (CIFAR-10/100) and real-world noise (Clothing1M and WebVision).

Accuracy comparison on CIFAR-10 with 40% noisy labels. GSS significantly reduces the gap to the reference upper limit compared to existing methods.

Key Results

  • Closing the Gap: On CIFAR-10 with 40% noise, GSS outperformed existing methods by up to 9.22%, moving significantly closer to the theoretical "clean data limit."
  • State-of-the-Art Performance: When integrated with semi-supervised learning (GSS-SSL), the method achieved top-tier results. On CIFAR-100 with 80% noise, it reached 55.04% accuracy, a substantial improvement over previous methods like DivideMix (45.82%).
  • Real-World Robustness: On the Clothing1M dataset (1 million images with ~61.54% label accuracy), GSS achieved 74.88% Top-1 accuracy, demonstrating its effectiveness against complex, non-random web noise.

Conclusion and Future Directions

GSS offers a proactive approach to noise by strategically manipulating gradients to keep model optimization on track. While highly effective, the strategy introduces some memory overhead to maintain the direction pool for massive datasets and requires hyperparameter tuning ($\lambda_1, \lambda_2$) to balance predicted versus random labels. Future work may focus on automating this parameter selection and further optimizing the memory footprint for web-scale applications. Overall, GSS provides a robust, pluggable enhancement for any noise-resistant deep learning pipeline.

Summary updated

BibTeX
@inproceedings{yu2023how,
  author = {Xiaotian Yu and Yang Jiang and Tianqi Shi and Zunlei Feng and Yuexuan Wang and Mingli Song and Li Sun},
  title = {{How to Prevent the Continuous Damage of Noises to Model Training?}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01160},
}
Figure